BINARY ATLAS 1 DATA FORMAT

This document describes the data format used for the release of O3a Atlas.

The release uses Mappable Vector Library (MVL) file to distribute data.

Every MVL file has the following architecture:

[PREAMBLE] .. [VECTOR] .. [ [VECTOR] ... ]  [POSTAMBLE]

The PREAMBLE starts with letter MVL0 to identify the file as MVL format, and contains information on data alignment and endianness. For this release the data is aligned on 64 byte boundary and is little-endian.

This is followed by VECTORs that carry data separated by optional padding. Each VECTOR has the following structure:

[HEADER] [DATA]

The 64-byte HEADER describes type and length of DATA and contains an optional 64-bit offset pointing to metadata:

typedef struct {
    LIBMVL_OFFSET64 length;
    int type;
    int reserved[11];
    LIBMVL_OFFSET64 metadata;
    } LIBMVL_VECTOR_HEADER;

The DATA is a plain array of length elements of given type. MVL supports integer and numeric types, character strings and a special LIBMVL_OFFSET64 type which is an unsigned 64-bit integer.

Knowing an offset from the beginning of the file uniquely identifies vectors stored in it. An array of offsets then acts as a list of vectors.

This allows to store data of arbitrary complexity in MVL file, whether a set of a tables similar to SQL database, a tree like structure, or a more complex data organization.

A metadata offset allows to associate additional information with a vector. A common use is to add a vector of "names" of the same length as an original vector, which can be used to retrieve data symbolically.

In order to retrieve data one needs to know offsets. They can be either provided externally (see table at the end of this document), or retrieved from other vectors in the MVL file.

The POSTAMBLE stores pointer to a directory, which is a named vector of offsets.

The low-level libMVL library is written in C and provides functions to write data into MVL file, access existing MVL files, parse metadata and perform database functions. See code and examples at:

https://github.com/volodya31415/libMVL

A higher level RMVL package for R integrates MVL files into R allowing easy analysis of data:

https://cran.r-project.org/package=RMVL

The examples provided with the data use RMVL. The R computing environment is open source:

https://www.r-project.org/

The RMVL package (and other R packages such as inline) are installed from within R by using the command:

install.packages("RMVL")

The data included in O3abinaryatlas1 consists of two tables (or in R terminology "data.frames") "parameters" and "skymaps":

These two tables have a common column "idx" which ties corresponding records.

"parameter" table columns description

• label - internal string labelling parameter record
• idx - index identifying corresponding records of parameter and skymaps table
• first_bin - starting bin of analyzed band in units of 1/14400 Hz
• band_start - first valid frequency of analyzed band
• band_stop - first frequency beyond analyzed band
• binary_cycle_start - first included phase of binary orbit at the epoch
• binary_cycle_stop - last included phase of binary orbit at the epoch
• resolution - this internal parameter has units of radians and controls spacing of sky grid

"skymaps" table columns description

• ul - worst-case 95% confidence level upper limit
• ul_circ - worst-case 95% confidence level upper limit on circularly polarized signals
• ul_avg - placeholder for population average proxy. PLEASE READ WARNING BELOW
• snr - maximum of signal-to-noise ratio in analyzed band
• snr_frequency - frequency where SNR maximum was achieved
• snr_iota - polarization parameter iota where SNR maximum was achieved
• snr_psi - polarization parameter psi where SNR maximum was achieved
• ra - Right Ascension in radians, J2000
• dec - Declination in radians, J2000
• stage - which stage analyzed the data (either 1 or 2)
• idx - index identifying corresponding records of parameter and skymaps table
• many more columns that are used to compute polarization specific upper limits (see spatial_index_example2.R)

WARNING: the column ul_avg is a placeholder for population average proxy. This column is a placeholder because determining correct values requires a lengthy simulation campaign. To prevent mistakes, the values stored there were set overly conservatively using population specific upper limits.

Each entry in the parameter table corresponds to a region in the parameter space that was analyzed on its own. The regions are created by selecting one of overlapping frequency bands and one of 10 phase regions of the binary orbit.

The atlas data is constructed in such a way, that if you are interested in a particular sky location and frequency, you need to find the entry in the skymaps table is closest in spherical distance and which frequency band covers frequency of interest. No corrections are needed for sky position mismatch, as it has already been accounted for.