This document describes the data format used for release of O3a Atlas.
The release uses Mappable Vector Library (MVL) file to distribute data.
Every MVL file has the following architecture:
[PREAMBLE] .. [VECTOR] .. [ [VECTOR] ... ] [POSTAMBLE]
The PREAMBLE starts with letter MVL0 to identify the file as MVL format, and contains information on data alignment and endianness. For this release the data is aligned on 64 byte boundary and is little-endian.
This is followed by VECTORs that carry data separated by optional padding. Each VECTOR has the following structure:
[HEADER] [DATA]
The 64-byte HEADER describes type and length of DATA and contains an optional 64-bit offset pointing to metadata:
typedef struct {
LIBMVL_OFFSET64 length;
int type;
int reserved[11];
LIBMVL_OFFSET64 metadata;
} LIBMVL_VECTOR_HEADER;
The DATA is a plain array of length elements of given type. MVL supports integer and numeric types, character strings and a special
LIBMVL_OFFSET64 type which is an unsigned 64-bit integer.
Knowing an offset from the beginning of the file uniquely identifies vectors stored in it. An array of offsets then acts as a list of vectors.
This allows to store data of arbitrary complexity in MVL file, whether a set of a tables similar to SQL database, a tree like structure, or a more complex data organization.
A metadata offset allows to associate additional information with a vector. A common use is to add a vector of "names" of the same length as an original vector, which can be used to retrieve data symbolically.
In order to retrieve data one needs to know offsets. They can be either provided externally (see table at the end of this document), or retrieved from other vectors in the MVL file.
The POSTAMBLE stores pointer to a directory, which is a named vector of offsets.
The low-level libMVL library is written in C and provides functions to write data into MVL file, access existing MVL files, parse metadata and perform database functions. See code and examples at:
https://github.com/volodya31415/libMVL
A higher level RMVL package for R integrates MVL files into R allowing easy analysis of data:
https://cran.r-project.org/package=RMVL
The examples provided with the data use RMVL. The R computing environment is open source:
The RMVL package (and other R packages such as inline) are installed from within R by using the command:
install.packages("RMVL")
The data included in O3a_atlas consists of two tables (or in R terminology "data.frames") "parameters" and "skymaps":
> parameters[1:5,]
label idx first_bin band_start band_stop resolution
1 0 1 288000 20.00333 20.06778 0.3743345
2 1 2 288648 20.04833 20.11278 0.3734956
3 10 3 294480 20.45333 20.51778 0.3661116
4 100 4 352800 24.50333 24.56778 0.3056788
5 1000 5 936000 65.00333 65.06778 0.1153215
> skymaps[1:5,]
ul ul_circ ul_avg ul_S snr snr_frequency snr_psi snr_iota ra dec stage idx rel_err
1 2.958927e-24 1.007307e-24 2.579873e-24 3.185131e-05 8.745301 20.01973 0.0000000 0.000000 0.0000000 0.3141593 02 1 0.015888453 ...
2 3.051866e-24 1.025739e-24 2.661516e-24 3.263957e-05 8.745301 20.01973 0.3926991 1.963495 0.3695991 0.3141593 02 1 0.013583064 ...
3 3.069451e-24 1.063987e-24 2.733632e-24 3.571220e-05 8.745301 20.01973 0.3926991 1.963495 0.1047198 0.6283185 02 1 0.003847361 ...
4 3.138739e-24 1.064705e-24 2.733632e-24 3.361250e-05 8.718230 20.01968 0.5235988 1.767146 0.3141593 0.6283185 02 1 0.013532281 ...
5 3.177858e-24 1.064705e-24 2.733632e-24 3.388693e-05 8.546866 20.01968 0.5235988 1.767146 0.5235988 0.6283185 02 1 0.009366274 ...
These two tables have a common column "idx" which ties corresponding records. Thus all entries shown above for skymaps table have idx=1 and, reading line 1 from parameter table, for these entries the frequency band spans 500.0033 Hz to 500.0500 Hz
"parameter" table columns description
label - internal string labelling parameter recordidx - index identifying corresponding records of parameter and skymaps tablefirst_bin - starting bin of analyzed band in units of 1/14400 Hzband_start - first valid frequency of analyzed bandband_stop - first frequency beyond analyzed bandresolution - this internal parameter has units of radians and controls spacing of sky grid"skymaps" table columns description
ul - worst-case 95% confidence level upper limitul_circ - worst-case 95% confidence level upper limit on circularly polarized signalsul_avg - placeholder for population average proxy. PLEASE READ WARNING BELOWsnr - maximum of signal-to-noise ratio in analyzed bandsnr_frequency - frequency where SNR maximum was achievedsnr_iota - polarization parameter iota where SNR maximum was achievedsnr_psi - polarization parameter psi where SNR maximum was achievedra - Right Ascension in radians, J2000dec - Declination in radians, J2000stage - which stage analyzed the data (either 1 or 2)idx - index identifying corresponding records of parameter and skymaps tablespatial_index_example2.R)WARNING: the column ul_avg is a placeholder for population average proxy. This column is a placeholder because determining correct values requires a simulation campaign that would delay the early release. To prevent mistakes, the values stored there were set overly conservatively using population specific upper limits.
Each entry in the parameter table corresponds to a region in the parameter space that was analyzed on its own. The regions are created by selecting one of overlapping frequency bands and one of 10 slices of the sky.
The atlas data is constructed in such a way, that if you are interested in a particular sky location and frequency, you need to find the entry in the skymaps table is closest in spherical distance and which frequency band covers frequency of interest. No corrections are needed for sky position mismatch, as it has already been accounted for.
O3a_2_atlas.mvl file path type length vector_offset
1 /parameters data.frame 6 0x0000000002119f00
2 /parameters/label string_vector 658000 0x000000000057d7a0
3 /parameters/idx double 658000 0x0000000000a82a80
4 /parameters/first_bin int32 658000 0x0000000000f87d40
5 /parameters/band_start double 658000 0x000000000120a6c0
6 /parameters/band_stop double 658000 0x000000000170f980
7 /parameters/resolution double 658000 0x0000000001c14c40
8 /skymaps data.frame 27 0x000000ba2f6f9340
9 /skymaps/ul float 7076327840 0x0000000002119f80
10 /skymaps/ul_circ float 7076327840 0x0000000699326640
11 /skymaps/ul_avg float 7076327840 0x0000000d30532d00
12 /skymaps/ul_S float 7076327840 0x00000013c773f3c0
13 /skymaps/snr float 7076327840 0x0000001a5e94ba80
14 /skymaps/snr_frequency double 7076327840 0x00000020f5b58140
15 /skymaps/snr_psi float 7076327840 0x0000002e23f70e80
16 /skymaps/snr_iota float 7076327840 0x00000034bb17d540
17 /skymaps/ra float 7076327840 0x0000003b52389c00
18 /skymaps/dec float 7076327840 0x00000041e95962c0
19 /skymaps/stage uint8 7076327840 0x00000048807a2980
20 /skymaps/idx double 7076327840 0x0000004a26425b60
21 /skymaps/rel_err float 7076327840 0x000000575483e8a0
22 /skymaps/unit float 7076327840 0x0000005deba4af60
23 /skymaps/pp float 7076327840 0x0000006482c57620
24 /skymaps/pc float 7076327840 0x0000006b19e63ce0
25 /skymaps/cc float 7076327840 0x00000071b10703a0
26 /skymaps/im_pc float 7076327840 0x000000784827ca60
27 /skymaps/pp2 float 7076327840 0x0000007edf489120
28 /skymaps/cc2 float 7076327840 0x00000085766957e0
29 /skymaps/pc2 float 7076327840 0x0000008c0d8a1ea0
30 /skymaps/ipp float 7076327840 0x00000092a4aae560
31 /skymaps/ipc float 7076327840 0x000000993bcbac20
32 /skymaps/icc float 7076327840 0x0000009fd2ec72e0
33 /skymaps/ppc float 7076327840 0x000000a66a0d39a0
34 /skymaps/pcc float 7076327840 0x000000ad012e0060
35 /skymaps/ppcc float 7076327840 0x000000b3984ec720
36 /skymaps_extent_index_idx mvl_index 6 0x000000ba30e09120
Vector types:
data.frame - 64-bit unsigned integer (LIBMVL_OFFSET64)string_vector - strings stored using libMVL LIBMVL_PACKED_LIST64 type,
see functions mvl_packed_list_get_entry and mvl_packed_list_get_entry_bytelength
or use RMVLfloat - 4-byte floating point numberdouble - 8-byte floating point numberuint8 - 1-byte unsigned integerint32 - 4-byte signed integer