B }dH@sdZddlZddlZddlZddlZddlmZddlmZddl m Z ddl Z ddl m ZddlmZd d lmZd d lmZd d lmZmZd d lmZdZedZedZedZedZedZ ed!d"dZ#ed!d"dZ$d5ddZ%ddZ&ddZ'ddZ(d d!Z)d"d#Z*d$d%Z+d&d'Z,e(d6d*d+Z-e(d7d-d.Z.e(d8d/d0Z/e(d9d1d2Z0e(d:d3d4Z1dS);a[Utilities for auto-discovery of GW data files. Automatic discovery of file paths for both LIGO and Virgo index solutions (``gwdatafind`` or ``ffl``, respectvely) is supported. The functions in this module are highly reliant on having local access to files (either directly, or via NFS, or CVMFS). Data discovery using the DataFind service requires the `gwdatafind` Python package (a dependency of ``gwpy``), and either the ``GW_DATAFIND_SERVER`` (or legacy ``LIGO_DATAFIND_SERVER``) environment variable to be set, or the ``host`` keyword must be passed to :func:`find_urls` and friends. Data discovery using the Virgo FFL system requires the ``FFLPATH`` environment variable to point to the directory containing FFL files, **or** the ``VIRGODATA`` environment variable to point to a directory containing an ``ffl` subdirectory, which contains FFL files. N) defaultdict)wraps)mock)segment)to_gps) ffldatafind)cache_segments) num_channelsiter_channel_names) file_pathz(Duncan Macleod z ^[A-Z][0-9]$z \A(.*_)?T\Zz \A(.*_)?M\Zz\A(.*_)?[Tt]rend\Zz^(?!.*_GRB\d{6}([A-Z])?$)z({})|)z$\A[A-Z]\d_HOFT_C\d\d(_T\d{7}_v\d)?\Zz \AV1Online\Zz'\AV1O[0-9]+([A-Z]+)?Repro[0-9]+[A-Z]+\Z)z_GRB\d{6}([A-Z])?\Zz_bck\Zz"\AT1200307_V4_EARLY_RECOLORED_V2\Zc Csx@dtfdtfdtfgD]&\}}||kr||rdt|fSqWxHtdtddtdtdtdtdi D]\}}| |rj|t|fSqjWdt|fS) zPrioritise the given GWF type based on its name or trend status. This is essentially an ad-hoc ordering function based on internal knowledge of how LIGO does GWF type naming. zm-trendzs-trendrrz [A-Z]\d_C ) LIGO_MINUTE_TREND_TYPELIGO_SECOND_TREND_TYPEVIRGO_SECOND_TREND_TYPEmatchlenHIGH_PRIORITY_TYPErecompileLOW_PRIORITY_TYPEitemssearch)ifoftypetrendZ trendnameZ trend_regexregprior"]/work/yifan.wang/ringdown/master-ringdown-env/lib/python3.7/site-packages/gwpy/io/datafind.py_type_priorityOs   r$c GsDx>|D]6}t|}y|jdkr$dSWqtk r:dSXqWdS)aDetermine whether any of the given files are on tape Parameters ---------- *files : `str` one or more paths to GWF files Returns ------- True/False : `bool` `True` if any of the files are determined to be on tape, otherwise `False` rTF)osstat st_blocksAttributeError)filespathr&r"r"r#on_tapems    r+cKsPtdstds|dr"tSy tWntk rJtdYnXtS)zReturn the appropriate GWDataFind-like API based on the environment This allows switching to the hacky `gwpy.io.ffldatafind` replacement module to enable a GWDataFind-like interface for direct FFL data discovery at Virgo. ZGWDATAFIND_SERVERZLIGO_DATAFIND_SERVERhostz4unknown datafind configuration, cannot discover data)r%getenvget gwdatafindr Z_get_ffl_basedirKeyError RuntimeError)Z datafind_kwr"r"r#_gwdatafind_modules     r2cstfdd}|S)zDecorate a function to see the right ``gwdatafind`` API. This exists only to allow on-the-fly replacing of the actual `gwdatafind` with :mod:`gwpy.io.ffldatafind` if it looks like we are trying to find data from FFL files. c s4tjjtf|jd<||SQRXdS)Nr/)rpatchdict __globals__r2)argskwargs)funcr"r#wrappedsz'_select_gwdatafind_mod..wrapped)r)r8r9r")r8r#_select_gwdatafind_modsr:c Cslddlm}t}xT|D]L}||}y||jd|jfWqtk rbtd|dYqXqW|S)zsz_rank_types..cSsi|]}t|t|f|qSr")r+r )rSr*r"r"r# sz_rank_types..cs|d f|dS)Nrrr")x)rankr"r#z_rank_types..)rTN)r=sort)rpathsrTr")rrXr# _rank_typess r]Ferrorc sZddlmt|ttfr |}n|g}fdd|D}dd|D} t|ttfrjt|} | d}nd} |dk rtt|}|dkr|dk r|dkrt }t tt } xFt |D]8\} } t | || d }x|D]}| |f| krqyt| |||d d }Wntttfk r wYnXt| || |}d}t| }y^xXt|D]L}|| krH|d 7}| ||||f|s| |||krHPqHWWn>tk r}ztd |d|dwWdd}~XYnX| | |f| sPqWqWt|||tfddD}|s:dd|D}t|ttfrN|S|t|S)aDFind the frametype(s) that hold data for a given channel Parameters ---------- channel : `str`, `~gwpy.detector.Channel` the channel to be found gpstime : `int`, optional target GPS time at which to find correct type frametype_match : `str`, optional regular expression to use for frametype `str` matching host : `str`, optional name of datafind host to use port : `int`, optional port on datafind host to use return_all : `bool`, optional, default: `False` return all found types, default is to return to 'best' match allow_tape : `bool`, optional, default: `False` do not test types whose frame files are stored on tape (not on spinning disk) on_gaps : `str`, optional action to take when gaps are discovered in datafind coverage, default: ``'error'``, i.e. don't match frametypes with gaps. Select ``'ignore'`` to ignore gaps, or ``'warn'`` to display warnings when gaps are found in a datafind `find_urls` query Returns ------- If a single name is given, and `return_all=False` (default): frametype : `str` name of best match frame type If a single name is given, and `return_all=True`: types : `list` of `str` the list of all matching frame types If multiple names are given, the above return types are wrapped into a `dict` of `(channel, type_or_list)` pairs. Examples -------- >>> from gwpy.io import datafind as io_datafind >>> io_datafind.find_frametype('H1:IMC-PWR_IN_OUTPUT', gpstime=1126259462) 'H1_R' >>> io_datafind.find_frametype('H1:IMC-PWR_IN_OUTPUT', gpstime=1126259462, ... return_all=True) ['H1_R', 'H1_C'] >>> io_datafind.find_frametype( ... ('H1:IMC-PWR_IN_OUTPUT', 'H1:OMC-DCPD_SUM_OUTPUT', ... 'H1:GDS-CALIB_STRAIN'), ... gpstime=1126259462, return_all=True))" {'H1:GDS-CALIB_STRAIN': ['H1_HOFT_C00'], 'H1:OMC-DCPD_SUM_OUTPUT': ['H1_R', 'H1_C'], 'H1:IMC-PWR_IN_OUTPUT': ['H1_R', 'H1_C']} r)r;csi|]}|j|qSr")rC)rSc)r;r"r#rVHsz"find_frametype..cSsi|]\}}||qSr"r")rSrTvalr"r"r#rVIsrNi*4)rrignore)rNrO on_missingrz!failed to read channels for type z: :cs&i|]}ttt|d|qS)r)listzip)rSrT)rr"r#rVscSsi|]\}}|d|qS)rr")rSrTr`r"r"r#rVs)r<r; isinstancerdtupler LigoSegmentintrGRB_TYPErr=rD find_types find_latestr1IOError IndexErrorrKrr appendpopwarningswarnr>rRkeysr]str)channelrNframetype_matchr,port return_allrOrEZchannelsnamesZ gpssegmentZsearchedrrtypesrr*ZgapsrBZnchannexcoutputr")r;rr#find_frametypesxF          r~Tc syt|||f||d||dStk rt|||f|d|d||dy$ttrdfddDSdStk rtd YnXYnXd S) aIntelligently select the best frametype from which to read this channel Parameters ---------- channel : `str`, `~gwpy.detector.Channel` the channel to be found start : `~gwpy.time.LIGOTimeGPS`, `float`, `str` GPS start time of period of interest, any input parseable by `~gwpy.time.to_gps` is fine end : `~gwpy.time.LIGOTimeGPS`, `float`, `str` GPS end time of period of interest, any input parseable by `~gwpy.time.to_gps` is fine host : `str`, optional name of datafind host to use port : `int`, optional port on datafind host to use frametype_match : `str`, optiona regular expression to use for frametype `str` matching allow_tape : `bool`, optional do not test types whose frame files are stored on tape (not on spinning disk) Returns ------- frametype : `str` the best matching frametype for the ``channel`` in the ``[start, end)`` interval Raises ------ ValueError if no valid frametypes are found Examples -------- >>> from gwpy.io.datafind import find_best_frametype >>> find_best_frametype('L1:GDS-CALIB_STRAIN', 1126259460, 1126259464) 'L1_HOFT_C00' r^)rNrvrOrEr,rwTra)rNrvrxrOrEr,rwcsi|]}|d|qS)rr")rSrT)ftoutr"r#rVsz'find_best_frametype..rz/Cannot find any valid frametypes for channel(s)N)r~r1rfr4rnrA)rustartendrvrOr,rwr")rr#find_best_frametypes 1     rc s(ttjfd|i|fdddS)zpFind the available data types for a given observatory. See also -------- gwdatafind.find_types rcst|dS)N)r)r$)rW) observatoryrr"r#rYrZzfind_types..)rT)sortedr/rk)rrrr7r")rrr#rksrkcKstj||||fd|i|S)zwFind the URLs of files of a given data type in a GPS interval. See also -------- gwdatafind.find_urls rE)r/rF)rrIrrrEr7r"r"r#rFsrFc Kst|r|d}yH|dk rDtt|}t||||dddd}ntj||ddd}Wn,ttfk rtd|d |YnXt |}|st |rt d |d |d ||S) zqFind the path of the latest file of a given data type. See also -------- gwdatafind.find_latest rNrra)rE)rbzno files found for -zLatest frame file for z- is on tape (pass allow_tape=True to force): ) SINGLE_IFO_OBSERVATORYrrirrFr/rlrnr1r r+rm)rrIrNrOr7r*r"r"r#rls0    rl)N)NNNNFFr^)NTNN)NN)r^)NF)2__doc__r%os.pathrrq collectionsr functoolsrZunittestrr/Z ligo.segmentsrrhtimerr rJr Zgwfr r utilsr __author__rrrrrrjformatrLrrr$r+r2r:rDrKrRr]r~rrkrFrlr"r"r"r#$s^                   5 @