O3a_3 ATLAS DATA FORMAT

This document describes the data format used for the release of O3a Atlas.

The release uses Mappable Vector Library (MVL) file to distribute data.

Every MVL file has the following architecture:

[PREAMBLE] .. [VECTOR] .. [ [VECTOR] ... ]  [POSTAMBLE]

The PREAMBLE starts with letter MVL0 to identify the file as MVL format, and contains information on data alignment and endianness. For this release the data is aligned on 64 byte boundary and is little-endian.

This is followed by VECTORs that carry data separated by optional padding. Each VECTOR has the following structure:

[HEADER] [DATA]

The 64-byte HEADER describes type and length of DATA and contains an optional 64-bit offset pointing to metadata:

typedef struct {
    LIBMVL_OFFSET64 length;
    int type;
    int reserved[11];
    LIBMVL_OFFSET64 metadata;
    } LIBMVL_VECTOR_HEADER;

The DATA is a plain array of length elements of given type. MVL supports integer and numeric types, character strings and a special LIBMVL_OFFSET64 type which is an unsigned 64-bit integer.

Knowing an offset from the beginning of the file uniquely identifies vectors stored in it. An array of offsets then acts as a list of vectors.

This allows to store data of arbitrary complexity in MVL file, whether a set of a tables similar to SQL database, a tree like structure, or a more complex data organization.

A metadata offset allows to associate additional information with a vector. A common use is to add a vector of "names" of the same length as an original vector, which can be used to retrieve data symbolically.

In order to retrieve data one needs to know offsets. They can be either provided externally (see table at the end of this document), or retrieved from other vectors in the MVL file.

The POSTAMBLE stores pointer to a directory, which is a named vector of offsets.

The low-level libMVL library is written in C and provides functions to write data into MVL file, access existing MVL files, parse metadata and perform database functions. See code and examples at:

https://github.com/volodya31415/libMVL

A higher level RMVL package for R integrates MVL files into R allowing easy analysis of data:

https://cran.r-project.org/package=RMVL

The examples provided with the data use RMVL. The R computing environment is open source:

https://www.r-project.org/

The RMVL package (and other R packages such as inline) are installed from within R by using the command:

install.packages("RMVL")

The data included in O3a3atlas consists of two tables (or in R terminology "data.frames") "parameters" and "skymaps":

> parameters[1:5,]
  label idx first_bin band_start band_stop resolution
1     0   1    288000   20.00333  20.06778  0.3743345
2     1   2    288648   20.04833  20.11278  0.3734956
3    10   3    294480   20.45333  20.51778  0.3661116
4   100   4    352800   24.50333  24.56778  0.3056788
5  1000   5    936000   65.00333  65.06778  0.1153215

> skymaps[1:5,]
            ul      ul_circ       ul_avg         ul_S      snr snr_frequency   snr_psi snr_iota        ra       dec stage idx     rel_err
1 2.958927e-24 1.007307e-24 2.579873e-24 3.185131e-05 8.745301      20.01973 0.0000000 0.000000 0.0000000 0.3141593    02   1 0.015888453 ...
2 3.051866e-24 1.025739e-24 2.661516e-24 3.263957e-05 8.745301      20.01973 0.3926991 1.963495 0.3695991 0.3141593    02   1 0.013583064 ...
3 3.069451e-24 1.063987e-24 2.733632e-24 3.571220e-05 8.745301      20.01973 0.3926991 1.963495 0.1047198 0.6283185    02   1 0.003847361 ...
4 3.138739e-24 1.064705e-24 2.733632e-24 3.361250e-05 8.718230      20.01968 0.5235988 1.767146 0.3141593 0.6283185    02   1 0.013532281 ...
5 3.177858e-24 1.064705e-24 2.733632e-24 3.388693e-05 8.546866      20.01968 0.5235988 1.767146 0.5235988 0.6283185    02   1 0.009366274 ...

These two tables have a common column "idx" which ties corresponding records. Thus all entries shown above for skymaps table have idx=1 and, reading line 1 from parameter table, for these entries the frequency band spans 20.00333 Hz to 20.06778 Hz

"parameter" table columns description

label - internal string labelling parameter record
idx - index identifying corresponding records of parameter and skymaps table
first_bin - starting bin of analyzed band in units of 1/14400 Hz
band_start - first valid frequency of analyzed band
band_stop - first frequency beyond analyzed band
resolution - this internal parameter has units of radians and controls spacing of sky grid

"skymaps" table columns description

ul - worst-case 95% confidence level upper limit
ul_circ - worst-case 95% confidence level upper limit on circularly polarized signals
ul_avg - placeholder for population average proxy. PLEASE READ WARNING BELOW
snr - maximum of signal-to-noise ratio in analyzed band
snr_frequency - frequency where SNR maximum was achieved
snr_iota - polarization parameter iota where SNR maximum was achieved
snr_psi - polarization parameter psi where SNR maximum was achieved
ra - Right Ascension in radians, J2000
dec - Declination in radians, J2000
stage - which stage analyzed the data (either 1 or 2)
idx - index identifying corresponding records of parameter and skymaps table
many more columns that are used to compute polarization specific upper limits (see spatial_index_example2.R)

WARNING: the column ul_avg is a placeholder for population average proxy. This column is a placeholder because determining correct values requires a simulation campaign that would delay the early release. To prevent mistakes, the values stored there were set overly conservatively using population specific upper limits.

Each entry in the parameter table corresponds to a region in the parameter space that was analyzed on its own. The regions are created by selecting one of overlapping frequency bands and one of 10 slices of the sky.

The atlas data is constructed in such a way, that if you are interested in a particular sky location and frequency, you need to find the entry in the skymaps table is closest in spherical distance and which frequency band covers frequency of interest. No corrections are needed for sky position mismatch, as it has already been accounted for.

Data layout in `O3a_3_atlas.mvl` file

                                   path          type      length      vector_offset
1                           /parameters    data.frame           6 0x0000000002593180
2                     /parameters/label string_vector      746920 0x000000000063b700
3                       /parameters/idx        double      746920 0x0000000000bee4a0
4                 /parameters/first_bin         int32      746920 0x00000000011a1220
5                /parameters/band_start        double      746920 0x000000000147a900
6                 /parameters/band_stop        double      746920 0x0000000001a2d680
7                /parameters/resolution        double      746920 0x0000000001fe0400
8                              /skymaps    data.frame          27 0x0000010d87b0ee00
9                           /skymaps/ul         float 10244100168 0x0000000002593200
10                     /skymaps/ul_circ         float 10244100168 0x000000098cbb7360
11                      /skymaps/ul_avg         float 10244100168 0x00000013171db4c0
12                        /skymaps/ul_S         float 10244100168 0x0000001ca17ff620
13                         /skymaps/snr         float 10244100168 0x000000262be23780
14               /skymaps/snr_frequency        double 10244100168 0x0000002fb64478e0
15                     /skymaps/snr_psi         float 10244100168 0x00000042cb08fb60
16                    /skymaps/snr_iota         float 10244100168 0x0000004c556b3cc0
17                          /skymaps/ra         float 10244100168 0x00000055dfcd7e20
18                         /skymaps/dec         float 10244100168 0x0000005f6a2fbf80
19                       /skymaps/stage         uint8 10244100168 0x00000068f49200e0
20                         /skymaps/idx        double 10244100168 0x0000006b572a9180
21                     /skymaps/rel_err         float 10244100168 0x0000007e6bef1400
22                        /skymaps/unit         float 10244100168 0x00000087f6515560
23                          /skymaps/pp         float 10244100168 0x0000009180b396c0
24                          /skymaps/pc         float 10244100168 0x0000009b0b15d820
25                          /skymaps/cc         float 10244100168 0x000000a495781980
26                       /skymaps/im_pc         float 10244100168 0x000000ae1fda5ae0
27                         /skymaps/pp2         float 10244100168 0x000000b7aa3c9c40
28                         /skymaps/cc2         float 10244100168 0x000000c1349edda0
29                         /skymaps/pc2         float 10244100168 0x000000cabf011f00
30                         /skymaps/ipp         float 10244100168 0x000000d449636060
31                         /skymaps/ipc         float 10244100168 0x000000ddd3c5a1c0
32                         /skymaps/icc         float 10244100168 0x000000e75e27e320
33                         /skymaps/ppc         float 10244100168 0x000000f0e88a2480
34                         /skymaps/pcc         float 10244100168 0x000000fa72ec65e0
35                        /skymaps/ppcc         float 10244100168 0x00000103fd4ea740
36            /skymaps_extent_index_idx     mvl_index           6 0x0000010d89427c20

Vector types:

data.frame - 64-bit unsigned integer (LIBMVL_OFFSET64)
string_vector - strings stored using libMVL LIBMVL_PACKED_LIST64 type, see functions mvl_packed_list_get_entry and mvl_packed_list_get_entry_bytelength or use RMVL
float - 4-byte floating point number
double - 8-byte floating point number
uint8 - 1-byte unsigned integer
int32 - 4-byte signed integer

O3a_3 ATLAS DATA FORMAT

Data layout in O3a_3_atlas.mvl file

Data layout in `O3a_3_atlas.mvl` file