Background Example ~~~~~~~~~~~~~~~~~~~~~~~ | | |image26| | In this example we do time lags analysis of H1, L1 detectors data on the so called O2 period. | NOTE : the test has been done with **SVN version 5099**. | All files used in this example are located in the cit cluster at : | `/home/waveburst/REPORTS/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1 `__ | In the appendix we report the list of commands used in the example and the shell screen output of each command. | The list of the main commands used to produce the background are reported in the `README `__. | PRE-PRODUCTION : Setup analysis ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Check environment _________________ | We suppose to have already installed the pipeline (see `CWB quick startup `__ or `How to install full CWB `__). | This means that if we simply write on our shell: .. code-block:: bash root -b -l | If you see something like this `root_shell <#root-shell>`__ then the installation is right (to exit from ROOT type ’.q’ & return) | Check all the dumped informations : libraries paths and versions. Create the Working Dir ______________________ Choose a name for your working dir and create it, for this case we choose: O2_BKG_L1H1_LF_WP10_tutorial_root6_run1: .. code-block:: bash cwb_mkdir O2_BKG_L1H1_LF_WP10_tutorial_root6_run1 | and the following dir/files have been created : `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/ `__ .. code-block:: bash config cwb configuration files input input data files (frame and veto list files) macro user macros & plugins condor condor files (dag, sub) log condor log files output output job files tmp dir for temporary job files data user output job files merge merge of output job files report dir for data to be pubblished on web Prepare needed files ____________________ For this analysis we need: - List of data **frame files** (mandatory) - **Data Quality** (not mandatory, but we know there are) - **Production configuration file** | For more information on how to obtain the frame files see: `How to make gw frame file list with gw_data_find `__ | For Data Quality see: `What are data quality? `__ - `How to create data quality file list `__ | For the complete list of parameters in the configuration file, see: `user_parameters.C `__ | For a detailed description how the job segments are created, see: `How job segments are created `__ | For a detailed description of the lags, see: `What are lags and how to use them `__ | For a detailed description of the slags, see: `What are super lags and how to use them `__ | We can copy the frame file and data quality lists from | `/home/waveburst/REPORTS/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/input `__ | to the **input** directory. .. code-block:: bash cd O2_BKG_L1H1_LF_WP10_tutorial_root6_run1 cp /home/waveburst/REPORTS/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/input/* input/. | Then we select the interesting parameters for production and put in the file : `config/user_parameters.C `__ The primary parameters are: - **Network**: H1L1 - **Job segment** lenght (600 seconds) - Number of **lags** (590 with 1 second step) - Number of **slags** (5) - List of **frame files** and **channel names** - **data quality** files | The full list of the parameters used by CWB is declared in the default parameter file `cwb_parameters.C `__ (most of parameters have a default value and should not be changed by the user). For a description of the such parameters see : `description of the parameters `__. The user_parameters.C contains only the user customization. | The final user parameters file used for this example is `user_parameters.C `__ | We can copy the configuration file from | `/home/waveburst/REPORTS/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/config `__ | to the **config** directory. .. code-block:: bash cd O2_BKG_L1H1_LF_WP10_tutorial_root6_run1 cp /home/waveburst/REPORTS/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/config/* config/. | Now we have the following files on the dir : `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/ `__ Define and Compile Plugin _________________________ | If the `config/user_parameters.C `__ contains a `user cwb plugin `__ then it must defined and compiled. | We use 2 system plugins: | - :cwb_library:`CWB_Plugin_QLveto.C` - ( `Documentation `__ ) - :cwb_library:`CWB_Plugin_Gating.C` - ( `Documentation `__ ) | are defined in the directory trunk plugin directory tools/cwb/plugins | | Since only one plugin can be defined in the user_parameters.C file we must merge the 2 plugins, we must use the `cwb_mplugin `__ command: .. code-block:: bash cwb_mplugin macro/CWB_Plugin_QLveto_Gating.C \ $HOME_CWB/plugins/CWB_Plugin_QLveto.C $HOME_CWB/plugins/CWB_Plugin_Gating.C | The output is : macro/CWB_Plugin_QLveto_Gating.C | Then compile it : .. code-block:: bash root -l -b macro/CWB_Plugin_QLveto_Gating.C++ Ignore errors like this: .. code-block:: bash Error: Function CWB_Plugin_Template() is not defined in current scope :0: *** Interpreter error recovered *** Some checks ___________ In principle we are ready to start the analysis. We can perform some checks to see if the setting are right using the `cwb_dump `__ commands. For instance we can be interested in the list of available segments than can be analyzed: .. code-block:: bash cwb_dump sjob | which create the file: | `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/report/dump/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.sjob `__ Or we can want the lag list: .. code-block:: bash cwb_dump lag | which create the file: | `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/report/dump/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.lag `__ or the coincidence times between the 2 detectors after cat1 and cat2 .. code-block:: bash cwb_dump dq | which create the files: | `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/report/dump/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.dq1 `__ | `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/report/dump/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.dq2 `__ The last final check can be to run in interactive mode the analysis on a job (the first, for instance) using `cwb_inet `__ command. If the job goes to end, the configuration is right. NOTE: This could take some time... .. code-block:: bash cwb_inet 2 | at the end you will see Job Terminated script and on data directory `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/data `__ | appear the file: .. code-block:: bash wave_1167537000_600_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1_slag0_lag0_590_job2.root | The file contains all the reconstructed event informations - root format (as a TTree) : \*.root | For a detailed description of the parameters reported in the root file see `trigger parameters `__ PRODUCTION - Run the analysis ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | In principle we can do analysis on the shell 3722 times using `cwb_inet `__ commands. If the cluster has a *batch system* you can use it for submitting jobs, one for each segment. cWB is optimized for condor, but it can be used also with LSF (Ex: CNAF cluster). Create Condor jobs __________________ Once ready, we can submit using `cwb_condor `__ command. First create the condor files: .. code-block:: bash cwb_condor create | which creates two files in condor directory `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/condor `__ | in details: .. code-block:: bash O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.dag condor job list O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.sub condor definitions cwb.sh shell script used by condor to make analysis ced.sh shell script used by condor to produce ced of loudest events and then we can submit job on cluster Submit Condor jobs __________________ .. code-block:: bash cwb_condor submit | Now, just wait to the jobs finish... | use condor command to check the job status, for example : | | to list job status : .. code-block:: bash condor_q user_name -nobatch to remove a job : .. code-block:: bash condor_rm job_id | (see job list to get job_id) | | It is possible to check the details of each job looking the condor log files under the log directory. | For example (job2) : | | `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/log/2_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.out `__ | `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/log/2_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.err `__ POST-PRODUCTION : Collection of results ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | Jobs produce files on the output directory `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/output `__ | Using default parameters, for each job there is a txt and a ROOT file. The ROOT file is considered for the production of final results, the txt files are stored if need to access in text mode. Merging files _____________ The final goal is to produce a web page showing some figure of merits. For this, it is useful to merge all the ROOT files in output directory in a single file, using `cwb_merge `__ command: .. code-block:: bash cwb_merge M1 '--nthreads 8' | M1 is a label identifing the merging. This command produces 3 files on merge directory `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/merge `__ | | in details: .. code-block:: bash merge_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.lst the list of merged files wave_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.root collection of trigger parameters live_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.root (only for background) collection of job informations (livetime, lags, ...) | Once obtained the merged file, we can produce the web page with the collection of figure of merits. | First of all we need a `user_pparameters.C `__ file where setting selection cuts and other information for the production of web pages. For this case the file is located here: `config/user_pparameters.C `__ Apply vetoes ____________ Note that we have selected the possibility to apply data quality and/or veto to produced triggers: .. code-block:: bash #define HVETO_VETO #define CAT2_VETO and the corresponding DQ/veto files: .. code-block:: bash dqfile vdqf[nvdqf] = { {"L1" ,"/home/waveburst/TUTORIALS/O2/DQ/O2_CO1_170104_170123/L1Cat2.txt", CWB_CAT2, 0., true, false}, {"H1" ,"/home/waveburst/TUTORIALS/O2/DQ/O2_CO1_170104_170123/H1Cat2.txt", CWB_CAT2, 0., true, false}, {"L1" ,"/home/waveburst/TUTORIALS/O2/DQ/O2_CO1_170104_170123/HVETO_L1_ANALYSIS3_SEGMENTS_MERGED.txt", CWB_HVETO, 0., false, false}, {"H1" ,"/home/waveburst/TUTORIALS/O2/DQ/O2_CO1_170104_170123/HVETO_H1_ANALYSIS3_SEGMENTS_MERGED.txt", CWB_HVETO, 0., false, false}, } | The DQ/veto application is done with `cwb_setveto `__ command. .. code-block:: bash cwb_setveto M1 | (be sure to use the same label as previous command). | Then on the merge dir `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/merge `__ there are two additional files: .. code-block:: bash cwb_user_parameters.C wave_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.root live_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.root merge_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.lst wave_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.V_cat2LH_hvetoLH.root live_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.V_cat2LH_hvetoLH.root Apply cuts __________ | Now we can apply `cwb_report `__ command. | This command applies thresholds on tree leaves to select events with certain characteristics defined by the user. | To know what is the right label: .. code-block:: bash cwb_merge list output: .. code-block:: bash -------------------- List of merge labels -------------------- - M1 - M1.V_cat2LH_hvetoLH Chosing the second one: .. code-block:: bash cwb_setcuts M1.V_cat2LH_hvetoLH "--tcuts bin1_cut --label bin1_cut" | The bin1_cut is label which identify the cuts rules and it is defined (`LH_BURST_LF_PP_Cuts.hh `__) in the user post-production file : `config/user_pparameters.C `__ Web page production ___________________ | Now we can produce web pages. In principle, we can decide to do it for both ROOT files in merge directory: applying or not applying DQ/veto. This depends on which label we use with `cwb_report `__ command. We select to produce results for the triggers applying DQ. | To know what is the right label: .. code-block:: bash cwb_merge list output: .. code-block:: bash -------------------- List of merge labels -------------------- - M1 - M1.V_cat2LH_hvetoLH - M1.V_cat2LH_hvetoLH.C_bin1_cut Chosing the third one: .. code-block:: bash cwb_report M1.V_cat2LH_hvetoLH.C_bin1_cut create This command create non-zero lag background report. If you want to create the zero-lag report do : .. code-block:: bash cwb_report M1.V_cat2LH_hvetoLH.C_bin1_cut create 0 0 If you want to create a specific lag_number slag_number report do : .. code-block:: bash cwb_report M1.V_cat2LH_hvetoLH.C_bin1_cut create lag_number slag_number The web page ____________ | The web page of the final report is : `report `__ | The web data files are produced under the directory : | `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/report/postprod/M1.V_cat2LH_hvetoLH.C_bin1_cut.R_rMRA_cat2_hveto_i0cc00_i0rho0_freq16_1024/data `__ | and can be view with a web browser at the following link : `web report link `__ | The script produces two more files under the condor directory : | `O2_BKG_L1H1_LF_WP10_tutorial_root6_run1/condor `__ .. code-block:: bash condor/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.V_cat2LH_hvetoLH.C_bin1_cut.R_rMRA_cat2_hveto_i0cc00_i0rho0_freq16_1024.ced.dag condor/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.V_cat2LH_hvetoLH.C_bin1_cut.R_rMRA_cat2_hveto_i0cc00_i0rho0_freq16_1024.ced.sub These are condor files necessary to produced the CED of the background loudest events (see the list on the bottom of web report page). To start the CED production do : .. code-block:: bash cwb_condor submit condor/O2_BKG_L1H1_LF_WP10_tutorial_root6_run1.M1.V_cat2LH_hvetoLH.C_bin1_cut.R_rMRA_cat2_hveto_i0cc00_i0rho0_freq16_1024.ced.dag | The output CED are stored under the "report/ced" directory and can be accessed using the links reported on the left of loudest event listed in the web page. | This is the CED of the loudest event : `CED link `__ How to make a CED of a detected event _____________________________________ | To production of the CED of a detected event can been done in batch or in interactive mode. | In the batch mode the CED are produced during the processing, to enable it the following parameters must be declared in the user_parameters.C configuration file : .. code-block:: bash cedDump = true; // dump ced enabled cedRHO = 4.0; // dump ced plots with rho>cedRHO To produce CED in interactive mode the command `cwb_inet `__ must be used : .. code-block:: bash cwb_inet job_number gps_time_of_the_first_detector true 0 lag_number | Ex : to produce the CED of the loudest event reported in the background page : `web report link `__ do: .. code-block:: bash cwb_inet 11385 0 ced 0 88 | where 11385 is the run number and 88 is the lag number | NOTE : the parameter must be set cedRHO less than the rho value of the detected event | The final ced is stored under the data directory. | To make ced visible from web do : .. code-block:: bash mv data/ced_1168134000_600_O2_BKG_L1H1_LF_WP10_tutorial_root6_run1_slag4_lag88_1_job11385 report/ced/ and can be view with a web browser at the following link : `web CED link `__ APPENDIX : Output on shell / Web pages ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We report here the output on shell for each command described in this section (see `README `__): .. toggle-header:: :header: **root logon**: Show/Hide Code .. literalinclude:: logs/bkg_examples_root_shell.log :language: bash .. toggle-header:: :header: **cwb_clonedir**: Show/Hide Code .. literalinclude:: logs/bkg_examples_cwb_clonedir_shell.log :language: bash .. toggle-header:: :header: **cwb_mkdir**: Show/Hide Code .. literalinclude:: logs/bkg_examples_cwb_mkdir_shell.log :language: bash .. toggle-header:: :header: **cwb_mkdir** Show/Hide Code .. literalinclude:: logs/bkg_examples_cwb_mkdir_result.log :language: bash .. toggle-header:: :header: **working dir** Show/Hide Code .. literalinclude:: logs/bkg_examples_dir_ready.log :language: bash .. toggle-header:: :header: **cwb_dump sjob**: Show/Hide Code .. literalinclude:: logs/bkg_examples_cwb_dump_sjob_shell.log :language: bash .. toggle-header:: :header: **compile plugin**: Show/Hide Code .. literalinclude:: logs/bkg_examples_compile_plugin_shell.log :language: bash .. toggle-header:: :header: **cwb_condor create** Show/Hide Code .. literalinclude:: logs/bkg_examples_cwb_condor_create_shell.log :language: bash .. toggle-header:: :header: **cwb_condor submit**: Show/Hide Code .. literalinclude:: logs/bkg_examples_cwb_condor_submit_shell.log :language: bash .. toggle-header:: :header: **cwb_merge**: Show/Hide Code .. literalinclude:: logs/bkg_examples_cwb_merge_shell.log :language: bash .. toggle-header:: :header: **cwb_setveto**: Show/Hide Code .. literalinclude:: logs/bkg_examples_cwb_setveto_shell.log :language: bash .. toggle-header:: :header: **cwb_setcuts**: Show/Hide Code .. literalinclude:: logs/bkg_examples_cwb_setcuts_shell.log :language: bash .. toggle-header:: :header: **cwb_report create**: Show/Hide Code .. literalinclude:: logs/bkg_examples_cwb_report_create_shell.log :language: bash | |