Generating concrete workflow 2023.06.11 18:10:12.482 GMT: [INFO] Planner launched in the following directory /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1 2023.06.11 18:10:12.490 GMT: [INFO] Planner invoked with following arguments --conf ./pegasus-properties.conf --cache /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map --output-sites local --sites local --staging-site local=local --cluster label,horizontal --cleanup inplace --relative-dir work -vvv --dir /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW --submit --forward nogrid gw.dax 2023.06.11 18:10:12.492 GMT: [CONFIG] Pegasus Properties set by the user 2023.06.11 18:10:12.492 GMT: [CONFIG] pegasus.catalog.replica.cache.asrc=true 2023.06.11 18:10:12.492 GMT: [CONFIG] pegasus.catalog.replica.dax.asrc=true 2023.06.11 18:10:12.493 GMT: [CONFIG] pegasus.catalog.workflow.amqp.url=amqp://friend:donatedata@msgs.pegasus.isi.edu:5672/prod/workflows 2023.06.11 18:10:12.493 GMT: [CONFIG] pegasus.dir.staging.mapper=Flat 2023.06.11 18:10:12.493 GMT: [CONFIG] pegasus.dir.storage.mapper=Replica 2023.06.11 18:10:12.493 GMT: [CONFIG] pegasus.dir.storage.mapper.replica=File 2023.06.11 18:10:12.493 GMT: [CONFIG] pegasus.dir.storage.mapper.replica.file=output.map 2023.06.11 18:10:12.493 GMT: [CONFIG] pegasus.dir.submit.mapper=Named 2023.06.11 18:10:12.494 GMT: [CONFIG] pegasus.home.bindir=/software/tools/pegasus/5.0/bin 2023.06.11 18:10:12.494 GMT: [CONFIG] pegasus.home.schemadir=/software/tools/pegasus/5.0/share/pegasus/schema 2023.06.11 18:10:12.494 GMT: [CONFIG] pegasus.home.sharedstatedir=/software/tools/pegasus/5.0/share/pegasus 2023.06.11 18:10:12.494 GMT: [CONFIG] pegasus.home.sysconfdir=/software/tools/pegasus/5.0/etc 2023.06.11 18:10:12.494 GMT: [CONFIG] pegasus.integrity.checking=none 2023.06.11 18:10:12.494 GMT: [CONFIG] pegasus.metrics.app=ligo-pycbc 2023.06.11 18:10:12.494 GMT: [CONFIG] pegasus.mode=production 2023.06.11 18:10:12.495 GMT: [CONFIG] pegasus.monitord.encoding=json 2023.06.11 18:10:12.495 GMT: [CONFIG] pegasus.register=False 2023.06.11 18:10:12.495 GMT: [CONFIG] pegasus.selector.replica=Regex 2023.06.11 18:10:12.495 GMT: [CONFIG] pegasus.selector.replica.regex.rank.1=file://(?!.*(cvmfs)).* 2023.06.11 18:10:12.495 GMT: [CONFIG] pegasus.selector.replica.regex.rank.2=file:///cvmfs/.* 2023.06.11 18:10:12.495 GMT: [CONFIG] pegasus.selector.replica.regex.rank.3=root://.* 2023.06.11 18:10:12.496 GMT: [CONFIG] pegasus.selector.replica.regex.rank.4=gsiftp://red-gridftp.unl.edu.* 2023.06.11 18:10:12.496 GMT: [CONFIG] pegasus.selector.replica.regex.rank.5=gridftp://.* 2023.06.11 18:10:12.496 GMT: [CONFIG] pegasus.selector.replica.regex.rank.6=.* 2023.06.11 18:10:12.496 GMT: [CONFIG] pegasus.transfer.bypass.input.staging=true 2023.06.11 18:10:12.496 GMT: [CONFIG] pegasus.transfer.links=true 2023.06.11 18:10:12.863 GMT: [INFO] event.pegasus.add.data-dependencies dax.id gw.dax-0 - STARTED 2023.06.11 18:10:12.863 GMT: [INFO] event.pegasus.add.data-dependencies dax.id gw.dax-0 (0.0 seconds) - FINISHED 2023.06.11 18:10:12.902 GMT: [DEBUG] Parsed DAX with following metrics {"compute_tasks":0,"dax_tasks":2,"dag_tasks":0,"total_tasks":2,"deleted_tasks":0,"dax_input_files":4,"dax_inter_files":0,"dax_output_files":0,"dax_total_files":4,"compute_jobs":0,"clustered_jobs":0,"si_tx_jobs":0,"so_tx_jobs":0,"inter_tx_jobs":0,"reg_jobs":0,"cleanup_jobs":0,"create_dir_jobs":0,"dax_jobs":2,"dag_jobs":0,"chmod_jobs":0,"total_jobs":2,"mDAXLabel":"gw.dax"} 2023.06.11 18:10:12.904 GMT: [CONFIG] Loading site catalog file /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/sites.yml 2023.06.11 18:10:12.904 GMT: [DEBUG] All sites will be loaded from the site catalog 2023.06.11 18:10:12.905 GMT: [DEBUG] event.pegasus.parse.site-catalog site-catalog.id /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/sites.yml - STARTED 2023.06.11 18:10:13.002 GMT: [DEBUG] event.pegasus.parse.site-catalog site-catalog.id /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/sites.yml (0.096 seconds) - FINISHED 2023.06.11 18:10:13.002 GMT: [DEBUG] Sites loaded are [osg, condorpool_shared, condorpool_symlink, condorpool_copy, local] 2023.06.11 18:10:13.004 GMT: [CONFIG] Set environment profile for local site PATH=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/bin/intel64:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin:/software/tools/pegasus/5.0/bin/:/work/ahnitz/projects/4ogc/env/bin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/bin/intel64:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin:/software/tools/pegasus/5.0/bin/:/work/ahnitz/projects/4ogc/env/bin:/software/tools/pegasus/5.0/bin/:/work/yifan.wang/lscsoft/opt/accomlal/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games 2023.06.11 18:10:13.004 GMT: [CONFIG] Set environment profile for local site PYTHONPATH=/software/tools/pegasus/5.0/lib/python3.7/dist-packages/:/software/tools/pegasus/5.0/lib/python3.7/dist-packages/:/software/tools/pegasus/5.0/lib/python3.7/dist-packages/:/work/yifan.wang/eccentricity/gitlab-eccsearch/waveform/ihes-teobresum/Python:/work/yifan.wang/eccentricity/gitlab-eccsearch//waveform/PyCBC-teobresums:/work/yifan.wang/lscsoft/src/TaylorF2e: 2023.06.11 18:10:13.004 GMT: [CONFIG] Constructed default site catalog entry for condorpool site condor 2023.06.11 18:10:13.092 GMT: [DEBUG] Mount Under Scratch Directories [/tmp, /var/tmp] 2023.06.11 18:10:13.093 GMT: [DEBUG] Style detected for site osg is class edu.isi.pegasus.planner.code.generator.condor.style.Condor 2023.06.11 18:10:13.093 GMT: [DEBUG] Style detected for site condorpool_shared is class edu.isi.pegasus.planner.code.generator.condor.style.Condor 2023.06.11 18:10:13.093 GMT: [DEBUG] Style detected for site condorpool is class edu.isi.pegasus.planner.code.generator.condor.style.Condor 2023.06.11 18:10:13.093 GMT: [DEBUG] Style detected for site condorpool_symlink is class edu.isi.pegasus.planner.code.generator.condor.style.Condor 2023.06.11 18:10:13.094 GMT: [DEBUG] Style detected for site condorpool_copy is class edu.isi.pegasus.planner.code.generator.condor.style.Condor 2023.06.11 18:10:13.094 GMT: [DEBUG] Style detected for site local is class edu.isi.pegasus.planner.code.generator.condor.style.Condor 2023.06.11 18:10:13.094 GMT: [DEBUG] Execution sites are [local] 2023.06.11 18:10:13.098 GMT: [CONFIG] Transformation Catalog Type used YAML TC 2023.06.11 18:10:13.098 GMT: [DEBUG] Ignoring error encountered while loading Transformation Catalog [1]: Unable to instantiate Transformation Catalog at edu.isi.pegasus.planner.catalog.transformation.TransformationFactory.loadInstance(TransformationFactory.java:164) [2]: edu.isi.pegasus.planner.catalog.transformation.impl.YAML caught java.lang.RuntimeException The File to be used as TC should be defined with the property pegasus.catalog.transformation.file at edu.isi.pegasus.planner.catalog.transformation.impl.YAML.connect(YAML.java:167) 2023.06.11 18:10:13.099 GMT: [DEBUG] Created a temporary transformation catalog backend /tmp/tc.1374778472659770564.txt 2023.06.11 18:10:13.100 GMT: [CONFIG] Transformation Catalog Type used Multiline Textual TC 2023.06.11 18:10:13.100 GMT: [CONFIG] Transformation Catalog File used /tmp/tc.1374778472659770564.txt 2023.06.11 18:10:13.102 GMT: [CONFIG] Data Configuration used for the workflow condorio 2023.06.11 18:10:13.103 GMT: [DEBUG] Directory to be created is /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/yifan.wang/pegasus/gw.dax/run0001 2023.06.11 18:10:13.104 GMT: [CONFIG] Metrics file will be written out to /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.metrics 2023.06.11 18:10:13.104 GMT: [CONFIG] The base submit directory for the workflow /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW 2023.06.11 18:10:13.105 GMT: [CONFIG] The relative submit directory for the workflow work 2023.06.11 18:10:13.105 GMT: [CONFIG] The relative execution directory for the workflow work 2023.06.11 18:10:13.106 GMT: [INFO] event.pegasus.stampede.events dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.111 GMT: [DEBUG] Written out stampede events for the abstract workflow to /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.static.bp 2023.06.11 18:10:13.111 GMT: [INFO] event.pegasus.stampede.events dax.id gw.dax-0 (0.005 seconds) - FINISHED 2023.06.11 18:10:13.112 GMT: [INFO] event.pegasus.refinement dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.118 GMT: [CONFIG] Proxy used for Replica Catalog is /tmp/x509up_u44039 2023.06.11 18:10:13.120 GMT: [DEBUG] [Replica Factory] Connect properties detected {proxy=/tmp/x509up_u44039, read.only=true, dax.asrc=true, cache.asrc=true} 2023.06.11 18:10:13.121 GMT: [DEBUG] [Replica Factory] Connect properties detected for implementor edu.isi.pegasus.planner.catalog.replica.impl.YAML -> {proxy=/tmp/x509up_u44039, read.only=true, dax.asrc=true, cache.asrc=true} 2023.06.11 18:10:13.123 GMT: [DEBUG] Problem while connecting with the Replica Catalog: Unable to connect to replica catalog implementation edu.isi.pegasus.planner.catalog.replica.impl.YAML with props {proxy=/tmp/x509up_u44039, read.only=true, dax.asrc=true, cache.asrc=true} 2023.06.11 18:10:13.123 GMT: [DEBUG] Setting property dagman.registration.maxjobs to 1 to set max jobs for registrations jobs category 2023.06.11 18:10:13.123 GMT: [DEBUG] Loading cache files: [/work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map] 2023.06.11 18:10:13.123 GMT: [INFO] event.pegasus.load.cache dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.123 GMT: [DEBUG] Loading file: /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map 2023.06.11 18:10:13.123 GMT: [DEBUG] [Replica Factory] Connect properties detected for implementor SimpleFile -> {file=/work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map, read.only=true, dax.asrc=true, cache.asrc=true} 2023.06.11 18:10:13.158 GMT: [DEBUG] Loaded 0 entry from file /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map 2023.06.11 18:10:13.159 GMT: [INFO] event.pegasus.load.cache dax.id gw.dax-0 (0.036 seconds) - FINISHED 2023.06.11 18:10:13.162 GMT: [DEBUG] Copied /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map to directory /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/catalogs 2023.06.11 18:10:13.163 GMT: [DEBUG] Copied /tmp/tc.1374778472659770564.txt to directory /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/catalogs 2023.06.11 18:10:13.163 GMT: [DEBUG] Copied /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/sites.yml to directory /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/catalogs 2023.06.11 18:10:13.164 GMT: [DEBUG] Set Default output replica catalog properties to {pegasus.catalog.replica.output.db.driver=sqlite, pegasus.catalog.replica.output.db.url=jdbc:sqlite:/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replicas.db, pegasus.catalog.replica.output=JDBCRC} 2023.06.11 18:10:13.164 GMT: [INFO] event.pegasus.check.cyclic-dependencies dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.164 GMT: [INFO] event.pegasus.check.cyclic-dependencies dax.id gw.dax-0 (0.0 seconds) - FINISHED 2023.06.11 18:10:13.165 GMT: [DEBUG] 0 entries found in cache of total 4 2023.06.11 18:10:13.165 GMT: [DEBUG] 0 entries found in previous submit dirs of total 4 2023.06.11 18:10:13.165 GMT: [DEBUG] 0 entries found in input directories of total 4 2023.06.11 18:10:13.165 GMT: [DEBUG] 4 entries found in abstract workflow replica store of total 4 2023.06.11 18:10:13.165 GMT: [DEBUG] 0 entries found in inherited replica store of total 4 2023.06.11 18:10:13.166 GMT: [DEBUG] 0 entries found in input replica catalog of total 4 2023.06.11 18:10:13.166 GMT: [DEBUG] 4 entries found in all replica sources of total 4 2023.06.11 18:10:13.166 GMT: [CONFIG] Data Reuse Scope for the workflow: full 2023.06.11 18:10:13.166 GMT: [DEBUG] Reducing the workflow 2023.06.11 18:10:13.166 GMT: [INFO] event.pegasus.reduce dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.167 GMT: [DEBUG] Jobs whose o/p files already exist 2023.06.11 18:10:13.167 GMT: [DEBUG] Job pegasus-plan_gw-main has no o/p files 2023.06.11 18:10:13.167 GMT: [DEBUG] Job pegasus-plan_gw-finalization has no o/p files 2023.06.11 18:10:13.167 GMT: [DEBUG] Jobs whose o/p files already exist - DONE 2023.06.11 18:10:13.168 GMT: [DEBUG] pegasus-plan_gw-main will not be deleted as not as child pegasus-plan_gw-finalization is not marked for deletion 2023.06.11 18:10:13.168 GMT: [INFO] Nodes/Jobs Deleted from the Workflow during reduction 2023.06.11 18:10:13.168 GMT: [INFO] Nodes/Jobs Deleted from the Workflow during reduction - DONE 2023.06.11 18:10:13.168 GMT: [INFO] event.pegasus.reduce dax.id gw.dax-0 (0.002 seconds) - FINISHED 2023.06.11 18:10:13.168 GMT: [INFO] event.pegasus.siteselection dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.172 GMT: [DEBUG] List of executions sites is [local] 2023.06.11 18:10:13.174 GMT: [DEBUG] Job pegasus-plan_gw-main will be mapped based on selector|hints profile key execution.site 2023.06.11 18:10:13.174 GMT: [DEBUG] Job pegasus-plan_gw-finalization will be mapped based on selector|hints profile key execution.site 2023.06.11 18:10:13.175 GMT: [DEBUG] Setting up site mapping for job pegasus-plan_gw-main 2023.06.11 18:10:13.175 GMT: [DEBUG] Job was mapped to pegasus-plan_gw-main to site local 2023.06.11 18:10:13.177 GMT: [DEBUG] Setting up site mapping for job pegasus-plan_gw-finalization 2023.06.11 18:10:13.177 GMT: [DEBUG] Job was mapped to pegasus-plan_gw-finalization to site local 2023.06.11 18:10:13.177 GMT: [INFO] event.pegasus.stampede.events dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.178 GMT: [DEBUG] Written out stampede metadata events for the mapped workflow to /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.static.bp 2023.06.11 18:10:13.178 GMT: [INFO] event.pegasus.stampede.events dax.id gw.dax-0 (0.001 seconds) - FINISHED 2023.06.11 18:10:13.178 GMT: [INFO] event.pegasus.siteselection dax.id gw.dax-0 (0.01 seconds) - FINISHED 2023.06.11 18:10:13.181 GMT: [DEBUG] User set mapper is not a stageable mapper. Loading a stageable mapper 2023.06.11 18:10:13.182 GMT: [DEBUG] Deployment of Worker Package needed 2023.06.11 18:10:13.196 GMT: [CONFIG] No Replica Registration Jobs will be created . 2023.06.11 18:10:13.200 GMT: [CONFIG] Transfer Implementation loaded for Stage-In [Python based Transfer Script] 2023.06.11 18:10:13.200 GMT: [CONFIG] Transfer Implementation loaded for symbolic linking Stage-In [Python based Transfer Script] 2023.06.11 18:10:13.201 GMT: [CONFIG] Transfer Implementation loaded for Inter Site [Python based Transfer Script] 2023.06.11 18:10:13.201 GMT: [CONFIG] Transfer Implementation loaded for Stage-Out [Python based Transfer Script] 2023.06.11 18:10:13.201 GMT: [DEBUG] Trying to get TCEntries for pegasus::worker on resource ALL of type STAGEABLE 2023.06.11 18:10:13.202 GMT: [DEBUG] System information for pegasus-worker-5.0.1-x86_64_deb_10.tar.gz is {arch=x86_64 os=linux osrelease=deb osversion=10} 2023.06.11 18:10:13.202 GMT: [DEBUG] Compute site sysinfo local {arch=x86_64 os=linux} 2023.06.11 18:10:13.202 GMT: [DEBUG] Worker Package Entry used for site local Logical Namespace : pegasus Logical Name : worker Version : null Resource Id : local Physical Name : file:///local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/pegasus-worker-5.0.1-x86_64_deb_10.tar.gz SysInfo : {arch=x86_64 os=linux} TYPE : STAGEABLE BYPASS : false Notifications: Container : null Compound Tx : null 2023.06.11 18:10:13.202 GMT: [DEBUG] Trying to get TCEntries for pegasus::worker on resource local of type STAGEABLE 2023.06.11 18:10:13.202 GMT: [DEBUG] Staging site for site local for worker package deployment - local 2023.06.11 18:10:13.202 GMT: [DEBUG] Trying to get TCEntries for pegasus::worker on resource ALL of type STAGEABLE 2023.06.11 18:10:13.203 GMT: [DEBUG] Selected entry Logical Namespace : pegasus Logical Name : worker Version : null Resource Id : local Physical Name : file:///local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/pegasus-worker-5.0.1-x86_64_deb_10.tar.gz SysInfo : {arch=x86_64 os=linux} TYPE : STAGEABLE BYPASS : false Notifications: Container : null Compound Tx : null 2023.06.11 18:10:13.203 GMT: [DEBUG] Creating a default TC entry for pegasus::transfer at site local 2023.06.11 18:10:13.203 GMT: [DEBUG] Remote Path set is pegasus-transfer 2023.06.11 18:10:13.203 GMT: [DEBUG] Trying to get TCEntries for pegasus::transfer on resource local of type INSTALLED 2023.06.11 18:10:13.204 GMT: [DEBUG] Entry constructed Logical Namespace : pegasus Logical Name : transfer Version : null Resource Id : local Physical Name : pegasus-transfer SysInfo : {arch=x86_64 os=linux} TYPE : INSTALLED BYPASS : false Notifications: Container : null Compound Tx : null 2023.06.11 18:10:13.204 GMT: [DEBUG] Creating a default TC entry for pegasus::kickstart at site local 2023.06.11 18:10:13.204 GMT: [DEBUG] Remote Path set is pegasus-kickstart 2023.06.11 18:10:13.204 GMT: [DEBUG] Trying to get TCEntries for pegasus::kickstart on resource local of type INSTALLED 2023.06.11 18:10:13.204 GMT: [DEBUG] Entry constructed Logical Namespace : pegasus Logical Name : kickstart Version : null Resource Id : local Physical Name : pegasus-kickstart SysInfo : {arch=x86_64 os=linux} TYPE : INSTALLED BYPASS : false Notifications: Container : null Compound Tx : null 2023.06.11 18:10:13.204 GMT: [DEBUG] Creating a default TC entry for pegasus::cleanup at site local 2023.06.11 18:10:13.204 GMT: [DEBUG] Remote Path set is pegasus-transfer 2023.06.11 18:10:13.204 GMT: [DEBUG] Trying to get TCEntries for pegasus::cleanup on resource local of type INSTALLED 2023.06.11 18:10:13.204 GMT: [DEBUG] Entry constructed Logical Namespace : pegasus Logical Name : cleanup Version : null Resource Id : local Physical Name : pegasus-transfer SysInfo : {arch=x86_64 os=linux} TYPE : INSTALLED BYPASS : false Notifications: Container : null Compound Tx : null 2023.06.11 18:10:13.204 GMT: [DEBUG] Creating a default TC entry for pegasus::seqexec at site local 2023.06.11 18:10:13.204 GMT: [DEBUG] Remote Path set is pegasus-cluster 2023.06.11 18:10:13.205 GMT: [DEBUG] Trying to get TCEntries for pegasus::seqexec on resource local of type INSTALLED 2023.06.11 18:10:13.205 GMT: [DEBUG] Entry constructed Logical Namespace : pegasus Logical Name : seqexec Version : null Resource Id : local Physical Name : pegasus-cluster SysInfo : {arch=x86_64 os=linux} TYPE : INSTALLED BYPASS : false Notifications: Container : null Compound Tx : null 2023.06.11 18:10:13.205 GMT: [DEBUG] Creating a default TC entry for pegasus::dirmanager at site local 2023.06.11 18:10:13.205 GMT: [DEBUG] Remote Path set is pegasus-transfer 2023.06.11 18:10:13.205 GMT: [DEBUG] Trying to get TCEntries for pegasus::dirmanager on resource local of type INSTALLED 2023.06.11 18:10:13.205 GMT: [DEBUG] Entry constructed Logical Namespace : pegasus Logical Name : dirmanager Version : null Resource Id : local Physical Name : pegasus-transfer SysInfo : {arch=x86_64 os=linux} TYPE : INSTALLED BYPASS : false Notifications: Container : null Compound Tx : null 2023.06.11 18:10:13.205 GMT: [DEBUG] Creating a default TC entry for pegasus::keg at site local 2023.06.11 18:10:13.205 GMT: [DEBUG] Remote Path set is pegasus-keg 2023.06.11 18:10:13.205 GMT: [DEBUG] Trying to get TCEntries for pegasus::keg on resource local of type INSTALLED 2023.06.11 18:10:13.205 GMT: [DEBUG] Entry constructed Logical Namespace : pegasus Logical Name : keg Version : null Resource Id : local Physical Name : pegasus-keg SysInfo : {arch=x86_64 os=linux} TYPE : INSTALLED BYPASS : false Notifications: Container : null Compound Tx : null 2023.06.11 18:10:13.206 GMT: [INFO] event.pegasus.cluster dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.208 GMT: [DEBUG] Adding job to graph pegasus-plan_gw-main 2023.06.11 18:10:13.208 GMT: [DEBUG] Adding job to graph pegasus-plan_gw-finalization 2023.06.11 18:10:13.208 GMT: [DEBUG] Adding parents for child gw-finalization 2023.06.11 18:10:13.210 GMT: [CONFIG] Partitioner loaded is Label Based Partitioning 2023.06.11 18:10:13.219 GMT: [CONFIG] Kickstart Stating Disabled Completely - false 2023.06.11 18:10:13.220 GMT: [CONFIG] Kickstart Stating Disabled Completely - false 2023.06.11 18:10:13.221 GMT: [CONFIG] Clusterer loaded is Topological based Vertical Clustering 2023.06.11 18:10:13.221 GMT: [INFO] Starting Graph Traversal 2023.06.11 18:10:13.222 GMT: [DEBUG] Adding to level 0 dummy 2023.06.11 18:10:13.222 GMT: [DEBUG] Adding to queue gw-main 2023.06.11 18:10:13.222 GMT: [DEBUG] Removed dummy 2023.06.11 18:10:13.222 GMT: [DEBUG] Adding to level 1 gw-main 2023.06.11 18:10:13.222 GMT: [DEBUG] Adding to queue gw-finalization 2023.06.11 18:10:13.222 GMT: [DEBUG] Removed gw-main 2023.06.11 18:10:13.222 GMT: [DEBUG] Adding to level 2 gw-finalization 2023.06.11 18:10:13.222 GMT: [DEBUG] Removed gw-finalization 2023.06.11 18:10:13.222 GMT: [INFO] Starting Graph Traversal - DONE 2023.06.11 18:10:13.222 GMT: [DEBUG] Partition is [gw-main] corresponding to label gw-main 2023.06.11 18:10:13.223 GMT: [DEBUG] Clustering jobs in partition ID1 [gw-main] 2023.06.11 18:10:13.223 GMT: [DEBUG] No clustering for partition ID1 2023.06.11 18:10:13.223 GMT: [DEBUG] Partition is [gw-finalization] corresponding to label gw-finalization 2023.06.11 18:10:13.223 GMT: [DEBUG] Clustering jobs in partition ID2 [gw-finalization] 2023.06.11 18:10:13.223 GMT: [DEBUG] No clustering for partition ID2 2023.06.11 18:10:13.223 GMT: [INFO] Determining relations between partitions 2023.06.11 18:10:13.224 GMT: [INFO] Determining relations between partitions - DONE 2023.06.11 18:10:13.224 GMT: [DEBUG] Adding job to graph pegasus-plan_gw-main 2023.06.11 18:10:13.224 GMT: [DEBUG] Adding job to graph pegasus-plan_gw-finalization 2023.06.11 18:10:13.224 GMT: [DEBUG] Adding parents for child gw-finalization 2023.06.11 18:10:13.225 GMT: [CONFIG] Partitioner loaded is Level Based Partitioning 2023.06.11 18:10:13.226 GMT: [CONFIG] Kickstart Stating Disabled Completely - false 2023.06.11 18:10:13.227 GMT: [CONFIG] Kickstart Stating Disabled Completely - false 2023.06.11 18:10:13.227 GMT: [CONFIG] Clusterer loaded is Horizontal Clustering 2023.06.11 18:10:13.227 GMT: [DEBUG] Adding to level 0 dummy 2023.06.11 18:10:13.227 GMT: [DEBUG] Adding to queue gw-main 2023.06.11 18:10:13.227 GMT: [DEBUG] Removed dummy 2023.06.11 18:10:13.227 GMT: [DEBUG] Adding to level 1 gw-main 2023.06.11 18:10:13.227 GMT: [DEBUG] Adding to queue gw-finalization 2023.06.11 18:10:13.227 GMT: [DEBUG] Removed gw-main 2023.06.11 18:10:13.227 GMT: [DEBUG] Partition ID1 is :[gw-main] 2023.06.11 18:10:13.227 GMT: [DEBUG] Clustering jobs in partition ID1 [gw-main] 2023.06.11 18:10:13.228 GMT: [DEBUG] Clustering jobs of type pegasus-pegasus-plan-5_0_1 2023.06.11 18:10:13.228 GMT: [DEBUG] No clustering of jobs mapped to execution site local 2023.06.11 18:10:13.228 GMT: [DEBUG] Adding to level 2 gw-finalization 2023.06.11 18:10:13.228 GMT: [DEBUG] Removed gw-finalization 2023.06.11 18:10:13.228 GMT: [DEBUG] Partition ID2 is :[gw-finalization] 2023.06.11 18:10:13.228 GMT: [DEBUG] Clustering jobs in partition ID2 [gw-finalization] 2023.06.11 18:10:13.228 GMT: [DEBUG] Clustering jobs of type pegasus-pegasus-plan-5_0_1 2023.06.11 18:10:13.228 GMT: [DEBUG] No clustering of jobs mapped to execution site local 2023.06.11 18:10:13.229 GMT: [DEBUG] Replacing {pegasus-plan_gw-main [] -> pegasus-plan_gw-finalization [],false} with {pegasus-plan_gw-main [] -> pegasus-plan_gw-finalization [],false}Add to set : true 2023.06.11 18:10:13.229 GMT: [DEBUG] All clustered jobs removed from the workflow 2023.06.11 18:10:13.229 GMT: [INFO] event.pegasus.cluster dax.id gw.dax-0 (0.023 seconds) - FINISHED 2023.06.11 18:10:13.229 GMT: [DEBUG] Initialising Replica Catalog for Planner Cache 2023.06.11 18:10:13.230 GMT: [DEBUG] [Replica Factory] Connect properties detected for implementor SimpleFile -> {output=JDBCRC, output.db.create=true, file=/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.getcache, read.only=true, dax.asrc=true, output.db.url=jdbc:sqlite:/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replicas.db, output.db.driver=sqlite, cache.asrc=true} 2023.06.11 18:10:13.230 GMT: [DEBUG] Initialising Replica Catalog for Planner Cache 2023.06.11 18:10:13.230 GMT: [DEBUG] [Replica Factory] Connect properties detected for implementor SimpleFile -> {output=JDBCRC, output.db.create=true, file=/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.putcache, read.only=true, dax.asrc=true, output.db.url=jdbc:sqlite:/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replicas.db, output.db.driver=sqlite, cache.asrc=true} 2023.06.11 18:10:13.230 GMT: [INFO] Grafting transfer nodes in the workflow 2023.06.11 18:10:13.230 GMT: [INFO] event.pegasus.generate.transfer-nodes dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.232 GMT: [DEBUG] Date Reuse Engine no longer tracks deleted leaf jobs. Returning empty list 2023.06.11 18:10:13.237 GMT: [CONFIG] No Replica Registration Jobs will be created . 2023.06.11 18:10:13.237 GMT: [DEBUG] Number of transfer jobs for -1 are 0 2023.06.11 18:10:13.238 GMT: [DEBUG] Number of transfer jobs for 0 are 1 2023.06.11 18:10:13.238 GMT: [DEBUG] Number of transfer jobs for 1 are 1 2023.06.11 18:10:13.239 GMT: [CONFIG] Transfer Implementation loaded for Stage-In [Python based Transfer Script] 2023.06.11 18:10:13.239 GMT: [CONFIG] Transfer Implementation loaded for symbolic linking Stage-In [Python based Transfer Script] 2023.06.11 18:10:13.239 GMT: [CONFIG] Transfer Implementation loaded for Inter Site [Python based Transfer Script] 2023.06.11 18:10:13.239 GMT: [CONFIG] Transfer Implementation loaded for Stage-Out [Python based Transfer Script] 2023.06.11 18:10:13.242 GMT: [DEBUG] Rank ( rank => 1 priority => 500 expr => file://(?!.*(cvmfs)).*) 2023.06.11 18:10:13.242 GMT: [DEBUG] Rank ( rank => 2 priority => 400 expr => file:///cvmfs/.*) 2023.06.11 18:10:13.242 GMT: [DEBUG] Rank ( rank => 3 priority => 300 expr => root://.*) 2023.06.11 18:10:13.242 GMT: [DEBUG] Rank ( rank => 4 priority => 200 expr => gsiftp://red-gridftp.unl.edu.*) 2023.06.11 18:10:13.242 GMT: [DEBUG] Rank ( rank => 5 priority => 100 expr => gridftp://.*) 2023.06.11 18:10:13.242 GMT: [DEBUG] Rank ( rank => 6 priority => 0 expr => .*) 2023.06.11 18:10:13.242 GMT: [CONFIG] [RegexReplicaSelector] User Provided Ranked regexes are [( rank => 1 priority => 500 expr => file://(?!.*(cvmfs)).*), ( rank => 2 priority => 400 expr => file:///cvmfs/.*), ( rank => 3 priority => 300 expr => root://.*), ( rank => 4 priority => 200 expr => gsiftp://red-gridftp.unl.edu.*), ( rank => 5 priority => 100 expr => gridftp://.*), ( rank => 6 priority => 0 expr => .*)] 2023.06.11 18:10:13.245 GMT: [DEBUG] [Replica Factory] Connect properties detected for implementor File -> {file=output.map, read.only=true} 2023.06.11 18:10:13.246 GMT: [CONFIG] Output Mapper loaded is [Replica Catalog Mapper] 2023.06.11 18:10:13.246 GMT: [DEBUG] Initialising Workflow Cache File in the Submit Directory 2023.06.11 18:10:13.246 GMT: [DEBUG] [Replica Factory] Connect properties detected for implementor FlushedCache -> {output=JDBCRC, output.db.create=true, file=/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.cache, dax.asrc=true, output.db.url=jdbc:sqlite:/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replicas.db, output.db.driver=sqlite, cache.asrc=true} 2023.06.11 18:10:13.248 GMT: [CONFIG] Transfer Refiner loaded is [Balanced Cluster Transfer Refiner( round robin distribution at file level)] 2023.06.11 18:10:13.248 GMT: [CONFIG] ReplicaSelector loaded is [Regex] 2023.06.11 18:10:13.248 GMT: [CONFIG] Submit Directory Mapper loaded is [Relative Submit Directory Mapper] 2023.06.11 18:10:13.248 GMT: [CONFIG] Staging Mapper loaded is [Flat Directory Staging Mapper] 2023.06.11 18:10:13.250 GMT: [DEBUG] SRM Server map is {} 2023.06.11 18:10:13.250 GMT: [DEBUG] SRM Server map is {} 2023.06.11 18:10:13.250 GMT: [DEBUG] Directory for job pegasus-plan_gw-main is . 2023.06.11 18:10:13.250 GMT: [DEBUG] 2023.06.11 18:10:13.250 GMT: [DEBUG] Job being traversed is pegasus-plan_gw-main 2023.06.11 18:10:13.250 GMT: [DEBUG] To be run at local 2023.06.11 18:10:13.250 GMT: [DEBUG] Parents of job:{} 2023.06.11 18:10:13.250 GMT: [DEBUG] Initialising Workflow Cache File for job pegasus-plan_gw-main 2023.06.11 18:10:13.251 GMT: [DEBUG] [Replica Factory] Connect properties detected for implementor FlushedCache -> {output=JDBCRC, output.db.create=true, file=/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/./pegasus-plan_gw-main.cache, dax.asrc=true, output.db.url=jdbc:sqlite:/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replicas.db, output.db.driver=sqlite, cache.asrc=true} 2023.06.11 18:10:13.251 GMT: [DEBUG] [RegexReplicaSelector] Selecting a pfn for lfn gw-main.dax at site local amongst gw-main.dax regex false -> {(/work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/gw-main.dax,{site=local}),} 2023.06.11 18:10:13.252 GMT: [DEBUG] Job Input files : Removed file gw-main.dax for job pegasus-plan_gw-main 2023.06.11 18:10:13.252 GMT: [DEBUG] Job Search Files : Removed file gw-main.dax for job pegasus-plan_gw-main 2023.06.11 18:10:13.252 GMT: [DEBUG] Set arguments for DAX job pegasus-plan_gw-main to -Dpegasus.dir.storage.mapper.replica.file=gw-main.map --basename gw-main --cluster label,horizontal --output-sites local --staging-site local=local --cache /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map --cleanup inplace -vvv /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/gw-main.dax 2023.06.11 18:10:13.254 GMT: [DEBUG] Directory for job pegasus-plan_gw-finalization is . 2023.06.11 18:10:13.254 GMT: [DEBUG] 2023.06.11 18:10:13.254 GMT: [DEBUG] Job being traversed is pegasus-plan_gw-finalization 2023.06.11 18:10:13.254 GMT: [DEBUG] To be run at local 2023.06.11 18:10:13.254 GMT: [DEBUG] Parents of job:{pegasus-plan_gw-main,} 2023.06.11 18:10:13.254 GMT: [DEBUG] Initialising Workflow Cache File for job pegasus-plan_gw-finalization 2023.06.11 18:10:13.255 GMT: [DEBUG] [Replica Factory] Connect properties detected for implementor FlushedCache -> {output=JDBCRC, output.db.create=true, file=/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/./pegasus-plan_gw-finalization.cache, dax.asrc=true, output.db.url=jdbc:sqlite:/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replicas.db, output.db.driver=sqlite, cache.asrc=true} 2023.06.11 18:10:13.255 GMT: [DEBUG] [RegexReplicaSelector] Selecting a pfn for lfn gw-finalization.dax at site local amongst gw-finalization.dax regex false -> {(/work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/gw-finalization.dax,{site=local}),} 2023.06.11 18:10:13.255 GMT: [DEBUG] Job Input files : Removed file gw-finalization.dax for job pegasus-plan_gw-finalization 2023.06.11 18:10:13.256 GMT: [DEBUG] Job Search Files : Removed file gw-finalization.dax for job pegasus-plan_gw-finalization 2023.06.11 18:10:13.256 GMT: [DEBUG] Set arguments for DAX job pegasus-plan_gw-finalization to -Dpegasus.dir.storage.mapper.replica.file=gw-finalization.map --basename gw-finalization --cluster label,horizontal --output-sites local --staging-site local=local --cache /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map --cleanup inplace -vvv /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/gw-finalization.dax 2023.06.11 18:10:13.257 GMT: [INFO] event.pegasus.generate.transfer-nodes dax.id gw.dax-0 (0.027 seconds) - FINISHED 2023.06.11 18:10:13.258 GMT: [DEBUG] Adding worker package deployment node for local 2023.06.11 18:10:13.258 GMT: [DEBUG] Skipping stage worker job for site local as worker package already in submit directory file:///local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/pegasus-worker-5.0.1-x86_64_deb_10.tar.gz 2023.06.11 18:10:13.258 GMT: [INFO] event.pegasus.generate.workdir-nodes dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.261 GMT: [DEBUG] Trying to get TCEntries for pegasus::dirmanager on resource local of type INSTALLED 2023.06.11 18:10:13.262 GMT: [DEBUG] Creating create dir node create_dir_gw.dax_0_local 2023.06.11 18:10:13.262 GMT: [DEBUG] Need to add edge create_dir_gw.dax_0_local -> pegasus-plan_gw-main 2023.06.11 18:10:13.262 GMT: [DEBUG] Adding node to the worfklow create_dir_gw.dax_0_local 2023.06.11 18:10:13.262 GMT: [INFO] event.pegasus.generate.workdir-nodes dax.id gw.dax-0 (0.004 seconds) - FINISHED 2023.06.11 18:10:13.262 GMT: [INFO] event.pegasus.generate.cleanup-nodes dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.267 GMT: [CONFIG] Setting property dagman.cleanup.maxjobs to 4 to set max jobs for cleanup jobs category 2023.06.11 18:10:13.267 GMT: [DEBUG] Number of sites 1 2023.06.11 18:10:13.267 GMT: [DEBUG] Site local count jobs = 3 2023.06.11 18:10:13.267 GMT: [DEBUG] * pegasus-plan_gw-main 2023.06.11 18:10:13.267 GMT: [DEBUG] * pegasus-plan_gw-finalization 2023.06.11 18:10:13.267 GMT: [DEBUG] * create_dir_gw.dax_0_local 2023.06.11 18:10:13.267 GMT: [DEBUG] local 3 2023.06.11 18:10:13.267 GMT: [DEBUG] Leaf jobs scheduled at site local are pegasus-plan_gw-main,pegasus-plan_gw-finalization,create_dir_gw.dax_0_local, 2023.06.11 18:10:13.267 GMT: [DEBUG] File gw-finalization.map will not be cleaned up for job pegasus-plan_gw-finalization 2023.06.11 18:10:13.267 GMT: [DEBUG] File gw-main.map will not be cleaned up for job pegasus-plan_gw-main 2023.06.11 18:10:13.268 GMT: [DEBUG] 2023.06.11 18:10:13.268 GMT: [INFO] For site: local number of files cleaned up - 0 2023.06.11 18:10:13.268 GMT: [DEBUG] CLEANUP LIST 2023.06.11 18:10:13.268 GMT: [INFO] event.pegasus.generate.cleanup-nodes dax.id gw.dax-0 (0.006 seconds) - FINISHED 2023.06.11 18:10:13.268 GMT: [INFO] Adding Leaf Cleanup Jobs dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.269 GMT: [DEBUG] Directory URL is a file url for site local [file:///atlas/user/scratch/yifan.wang/pycbc-tmp_qw7u8m6k/local-site-scratch/work] 2023.06.11 18:10:13.269 GMT: [DEBUG] Trying to get TCEntries for pegasus::cleanup on resource local of type INSTALLED 2023.06.11 18:10:13.270 GMT: [DEBUG] Creating remove directory node cleanup_gw.dax_0_local 2023.06.11 18:10:13.270 GMT: [DEBUG] Need to add edge for DAX|DAG job pegasus-plan_gw-finalization -> cleanup_gw.dax_0_local 2023.06.11 18:10:13.270 GMT: [DEBUG] Need to add edge pegasus-plan_gw-finalization -> cleanup_gw.dax_0_local 2023.06.11 18:10:13.270 GMT: [DEBUG] Need to add edge for DAX|DAG job pegasus-plan_gw-main -> cleanup_gw.dax_0_local 2023.06.11 18:10:13.270 GMT: [DEBUG] Adding node to the worklfow cleanup_gw.dax_0_local 2023.06.11 18:10:13.270 GMT: [INFO] Adding Leaf Cleanup Jobs dax.id gw.dax-0 (0.002 seconds) - FINISHED 2023.06.11 18:10:13.272 GMT: [INFO] event.pegasus.refinement dax.id gw.dax-0 (0.16 seconds) - FINISHED 2023.06.11 18:10:13.303 GMT: [DEBUG] Condor Version as string 8.8.9 2023.06.11 18:10:13.303 GMT: [DEBUG] Condor Version detected is 80809 2023.06.11 18:10:13.303 GMT: [INFO] Generating codes for the executable workflow 2023.06.11 18:10:13.304 GMT: [INFO] event.pegasus.code.generation dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.304 GMT: [CONFIG] Kickstart Stating Disabled Completely - false 2023.06.11 18:10:13.304 GMT: [DEBUG] event.pegasus.code.generation dax.id gw.dax-0 - STARTED 2023.06.11 18:10:13.329 GMT: [DEBUG] Condor Version as string 8.8.9 2023.06.11 18:10:13.330 GMT: [DEBUG] Condor Version detected is 80809 2023.06.11 18:10:13.330 GMT: [CONFIG] Kickstart Stating Disabled Completely - false 2023.06.11 18:10:13.331 GMT: [DEBUG] Applying priority of 800 to create_dir_gw.dax_0_local 2023.06.11 18:10:13.361 GMT: [DEBUG] Mount Under Scratch Directories [/tmp, /var/tmp] 2023.06.11 18:10:13.367 GMT: [CONFIG] Kickstart Stating Disabled Completely - false 2023.06.11 18:10:13.371 GMT: [DEBUG] Trying to get TCEntries for pegasus::kickstart on resource local of type INSTALLED 2023.06.11 18:10:13.372 GMT: [DEBUG] Postscript constructed is /software/tools/pegasus/5.0/bin/pegasus-exitcode 2023.06.11 18:10:13.372 GMT: [WARNING] Removing unsupported key request_cpus in local universe for job create_dir_gw.dax_0_local 2023.06.11 18:10:13.372 GMT: [WARNING] Removing unsupported key request_memory in local universe for job create_dir_gw.dax_0_local 2023.06.11 18:10:13.373 GMT: [DEBUG] Unquoted arguments are pegasus-kickstart -n pegasus::dirmanager -N null -i - -R local -L gw.dax -T 2023-06-11T18:10:08+00:00 pegasus-transfer 2023.06.11 18:10:13.373 GMT: [DEBUG] Quoted arguments are "pegasus-kickstart -n pegasus::dirmanager -N null -i - -R local -L gw.dax -T 2023-06-11T18:10:08+00:00 pegasus-transfer " 2023.06.11 18:10:13.374 GMT: [DEBUG] Written Submit file : /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/./create_dir_gw.dax_0_local.sub 2023.06.11 18:10:13.374 GMT: [DEBUG] Applying priority of 10 to pegasus-plan_gw-main 2023.06.11 18:10:13.374 GMT: [DEBUG] Generating code for DAX job pegasus-plan_gw-main 2023.06.11 18:10:13.374 GMT: [DEBUG] Arguments passed to SUBDAX Generator are -Dpegasus.dir.storage.mapper.replica.file=gw-main.map --basename gw-main --cluster label,horizontal --output-sites local --staging-site local=local --cache /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map --cleanup inplace -vvv /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/gw-main.dax 2023.06.11 18:10:13.376 GMT: [DEBUG] Retrieving Metadata from the DAX file /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/gw-main.dax 2023.06.11 18:10:22.078 GMT: [DEBUG] Submit directory in sub dax specified is ./gw-main.dax_gw-main 2023.06.11 18:10:22.079 GMT: [DEBUG] Base Submit directory for inner workflow set to /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW 2023.06.11 18:10:22.079 GMT: [DEBUG] Relative Submit Directory for inner workflow set to work/././gw-main.dax_gw-main 2023.06.11 18:10:22.079 GMT: [DEBUG] Submit directory for inner workflow set to /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/././gw-main.dax_gw-main 2023.06.11 18:10:22.079 GMT: [DEBUG] Setting list of execution sites to the same as outer workflow 2023.06.11 18:10:22.079 GMT: [DEBUG] Submit Directory for SUB DAX is /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/././gw-main.dax_gw-main 2023.06.11 18:10:22.079 GMT: [DEBUG] Relative Execution Directory for SUB DAX is work/./gw-main.dax_gw-main 2023.06.11 18:10:22.081 GMT: [DEBUG] Trying to get TCEntries for pegasus::pegasus-plan on resource local of type INSTALLED 2023.06.11 18:10:22.081 GMT: [DEBUG] Constructing the default path to the pegasus-plan 2023.06.11 18:10:22.082 GMT: [DEBUG] pegasus-plan invocation for job pegasus-plan_gw-main determined to be /software/tools/pegasus/5.0/bin/pegasus-plan -Dpegasus.log.*=/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/pegasus-plan_gw-main.pre.log -Dpegasus.workflow.root.uuid=85e401d4-463b-45ff-91e7-7e875a962581 -Dpegasus.dir.storage.mapper.replica.file=gw-main.map --conf /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/pegasus.16970775812990819111.properties --dir /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW --relative-dir work/./gw-main.dax_gw-main --relative-submit-dir work/././gw-main.dax_gw-main --basename gw-main --sites local --staging-site local=local, --cache /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/./pegasus-plan_gw-main.cache,/work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map --inherited-rc-files /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replica.store --cluster label,horizontal --output-sites local --cleanup none --verbose --verbose --verbose --deferred /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/gw-main.dax 2023.06.11 18:10:22.082 GMT: [DEBUG] Basename prefix for the sub workflow is gw-main 2023.06.11 18:10:22.082 GMT: [DEBUG] Cache File for the sub workflow is /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/././gw-main.dax_gw-main/gw-main.cache 2023.06.11 18:10:22.082 GMT: [DEBUG] Trying to get TCEntries for condor::dagman on resource local of type INSTALLED 2023.06.11 18:10:22.082 GMT: [DEBUG] condor::dagman not catalogued in the Transformation Catalog. Trying to construct from the Site Catalog 2023.06.11 18:10:22.082 GMT: [DEBUG] DAGMan not catalogued in the Transformation Catalog or the Site Catalog. Trying to construct from the environment 2023.06.11 18:10:22.082 GMT: [DEBUG] Constructing path to dagman on basis of env variable CONDOR_LOCATION 2023.06.11 18:10:22.083 GMT: [DEBUG] Number of Resuce retries 999 2023.06.11 18:10:22.083 GMT: [DEBUG] Constructing arguments to dagman in 7.1.0 and later style 2023.06.11 18:10:22.085 GMT: [CONFIG] Kickstart Stating Disabled Completely - false 2023.06.11 18:10:22.086 GMT: [DEBUG] Setting job pegasus-plan_gw-main.pre to run via No container wrapping 2023.06.11 18:10:22.086 GMT: [DEBUG] Trying to get TCEntries for pegasus::kickstart on resource local of type INSTALLED 2023.06.11 18:10:22.086 GMT: [DEBUG] Trying to get TCEntries for pegasus::transfer on resource local of type INSTALLED 2023.06.11 18:10:22.090 GMT: [DEBUG] Unquoted arguments are -p 0 -f -l . -Notification never -Debug 3 -Lockfile gw-main.dag.lock -Dag gw-main.dag -AllowVersionMismatch -AutoRescue 1 -DoRescueFrom 0 -MaxPre 1 2023.06.11 18:10:22.091 GMT: [DEBUG] Quoted arguments are " -p 0 -f -l . -Notification never -Debug 3 -Lockfile gw-main.dag.lock -Dag gw-main.dag -AllowVersionMismatch -AutoRescue 1 -DoRescueFrom 0 -MaxPre 1" 2023.06.11 18:10:22.091 GMT: [DEBUG] Written Submit file : /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/./pegasus-plan_gw-main.sub 2023.06.11 18:10:22.091 GMT: [DEBUG] Applying priority of 20 to pegasus-plan_gw-finalization 2023.06.11 18:10:22.091 GMT: [DEBUG] Generating code for DAX job pegasus-plan_gw-finalization 2023.06.11 18:10:22.091 GMT: [DEBUG] Arguments passed to SUBDAX Generator are -Dpegasus.dir.storage.mapper.replica.file=gw-finalization.map --basename gw-finalization --cluster label,horizontal --output-sites local --staging-site local=local --cache /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map --cleanup inplace -vvv /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/gw-finalization.dax 2023.06.11 18:10:22.092 GMT: [DEBUG] Retrieving Metadata from the DAX file /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/gw-finalization.dax 2023.06.11 18:10:22.096 GMT: [DEBUG] Submit directory in sub dax specified is ./gw-finalization.dax_gw-finalization 2023.06.11 18:10:22.096 GMT: [DEBUG] Base Submit directory for inner workflow set to /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW 2023.06.11 18:10:22.096 GMT: [DEBUG] Relative Submit Directory for inner workflow set to work/././gw-finalization.dax_gw-finalization 2023.06.11 18:10:22.096 GMT: [DEBUG] Submit directory for inner workflow set to /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/././gw-finalization.dax_gw-finalization 2023.06.11 18:10:22.096 GMT: [DEBUG] Setting list of execution sites to the same as outer workflow 2023.06.11 18:10:22.096 GMT: [DEBUG] Parent DAX Jobs Transient RC's are [/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/././gw-main.dax_gw-main/gw-main.cache] 2023.06.11 18:10:22.096 GMT: [DEBUG] Submit Directory for SUB DAX is /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/././gw-finalization.dax_gw-finalization 2023.06.11 18:10:22.096 GMT: [DEBUG] Relative Execution Directory for SUB DAX is work/./gw-finalization.dax_gw-finalization 2023.06.11 18:10:22.097 GMT: [DEBUG] Trying to get TCEntries for pegasus::pegasus-plan on resource local of type INSTALLED 2023.06.11 18:10:22.097 GMT: [DEBUG] Constructing the default path to the pegasus-plan 2023.06.11 18:10:22.097 GMT: [DEBUG] pegasus-plan invocation for job pegasus-plan_gw-finalization determined to be /software/tools/pegasus/5.0/bin/pegasus-plan -Dpegasus.log.*=/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/pegasus-plan_gw-finalization.pre.log -Dpegasus.workflow.root.uuid=85e401d4-463b-45ff-91e7-7e875a962581 -Dpegasus.dir.storage.mapper.replica.file=gw-finalization.map --conf /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/pegasus.16970775812990819111.properties --dir /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW --relative-dir work/./gw-finalization.dax_gw-finalization --relative-submit-dir work/././gw-finalization.dax_gw-finalization --basename gw-finalization --sites local --staging-site local=local, --cache /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/./pegasus-plan_gw-finalization.cache,/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/././gw-main.dax_gw-main/gw-main.cache,/work/yifan.wang/search-high-spin/inj-t11-detmass1p4/prod.map --inherited-rc-files /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replica.store --cluster label,horizontal --output-sites local --cleanup none --verbose --verbose --verbose --deferred /work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1/gw-finalization.dax 2023.06.11 18:10:22.097 GMT: [DEBUG] Basename prefix for the sub workflow is gw-finalization 2023.06.11 18:10:22.097 GMT: [DEBUG] Cache File for the sub workflow is /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/././gw-finalization.dax_gw-finalization/gw-finalization.cache 2023.06.11 18:10:22.097 GMT: [DEBUG] Trying to get TCEntries for condor::dagman on resource local of type INSTALLED 2023.06.11 18:10:22.097 GMT: [DEBUG] condor::dagman not catalogued in the Transformation Catalog. Trying to construct from the Site Catalog 2023.06.11 18:10:22.097 GMT: [DEBUG] DAGMan not catalogued in the Transformation Catalog or the Site Catalog. Trying to construct from the environment 2023.06.11 18:10:22.097 GMT: [DEBUG] Constructing path to dagman on basis of env variable CONDOR_LOCATION 2023.06.11 18:10:22.097 GMT: [DEBUG] Number of Resuce retries 999 2023.06.11 18:10:22.097 GMT: [DEBUG] Constructing arguments to dagman in 7.1.0 and later style 2023.06.11 18:10:22.098 GMT: [DEBUG] Setting job pegasus-plan_gw-finalization.pre to run via No container wrapping 2023.06.11 18:10:22.098 GMT: [DEBUG] Trying to get TCEntries for pegasus::kickstart on resource local of type INSTALLED 2023.06.11 18:10:22.098 GMT: [DEBUG] Trying to get TCEntries for pegasus::transfer on resource local of type INSTALLED 2023.06.11 18:10:22.099 GMT: [DEBUG] Unquoted arguments are -p 0 -f -l . -Notification never -Debug 3 -Lockfile gw-finalization.dag.lock -Dag gw-finalization.dag -AllowVersionMismatch -AutoRescue 1 -DoRescueFrom 0 -MaxPre 1 2023.06.11 18:10:22.099 GMT: [DEBUG] Quoted arguments are " -p 0 -f -l . -Notification never -Debug 3 -Lockfile gw-finalization.dag.lock -Dag gw-finalization.dag -AllowVersionMismatch -AutoRescue 1 -DoRescueFrom 0 -MaxPre 1" 2023.06.11 18:10:22.099 GMT: [DEBUG] Written Submit file : /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/./pegasus-plan_gw-finalization.sub 2023.06.11 18:10:22.099 GMT: [DEBUG] Applying priority of 1000 to cleanup_gw.dax_0_local 2023.06.11 18:10:22.100 GMT: [DEBUG] Trying to get TCEntries for pegasus::kickstart on resource local of type INSTALLED 2023.06.11 18:10:22.100 GMT: [DEBUG] Postscript constructed is /software/tools/pegasus/5.0/bin/pegasus-exitcode 2023.06.11 18:10:22.100 GMT: [WARNING] Removing unsupported key request_cpus in local universe for job cleanup_gw.dax_0_local 2023.06.11 18:10:22.100 GMT: [WARNING] Removing unsupported key request_memory in local universe for job cleanup_gw.dax_0_local 2023.06.11 18:10:22.100 GMT: [DEBUG] Unquoted arguments are pegasus-kickstart -n pegasus::cleanup -N null -i - -R local -L gw.dax -T 2023-06-11T18:10:08+00:00 pegasus-transfer 2023.06.11 18:10:22.100 GMT: [DEBUG] Quoted arguments are "pegasus-kickstart -n pegasus::cleanup -N null -i - -R local -L gw.dax -T 2023-06-11T18:10:08+00:00 pegasus-transfer " 2023.06.11 18:10:22.100 GMT: [DEBUG] Written Submit file : /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/./cleanup_gw.dax_0_local.sub 2023.06.11 18:10:22.100 GMT: [DEBUG] event.pegasus.code.generation dax.id gw.dax-0 (8.796 seconds) - FINISHED 2023.06.11 18:10:22.101 GMT: [DEBUG] Written Dag File : gw.dax-0.dag.tmp 2023.06.11 18:10:22.101 GMT: [DEBUG] Writing out the DOT file 2023.06.11 18:10:22.110 GMT: [DEBUG] Written out notifications to gw.dax-0.notify 2023.06.11 18:10:22.110 GMT: [DEBUG] Writing out the DAX Replica Store to file /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replica.store 2023.06.11 18:10:22.110 GMT: [DEBUG] [Replica Factory] Connect properties detected for implementor SimpleFile -> {output=JDBCRC, output.db.create=true, file=/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replica.store, dax.asrc=true, output.db.url=jdbc:sqlite:/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replicas.db, output.db.driver=sqlite, cache.asrc=true} 2023.06.11 18:10:22.111 GMT: [DEBUG] Written out dax replica store to gw.dax-0.replica.store 2023.06.11 18:10:22.112 GMT: [DEBUG] Written out stampede events for the executable workflow to /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.static.bp 2023.06.11 18:10:22.118 GMT: [DEBUG] Proxy whose DN will be logged in the braindump file /tmp/x509up_u44039 2023.06.11 18:10:22.160 GMT: [DEBUG] Unable to determine GRID DN class org.globus.gsi.gssapi.GlobusGSSException: Defective credential detected [Caused by: proxy not found] 2023.06.11 18:10:22.189 GMT: [DEBUG] Written out braindump to /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/braindump.yml 2023.06.11 18:10:22.189 GMT: [DEBUG] Renamed temporary dag file to : /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.dag 2023.06.11 18:10:22.191 GMT: [DEBUG] Executing /usr/bin/condor_submit_dag -append executable=/software/tools/pegasus/5.0/bin/pegasus-dagman -no_submit -MaxPre 1 -MaxPost 20 -append +pegasus_wf_uuid="85e401d4-463b-45ff-91e7-7e875a962581" -append +pegasus_root_wf_uuid="85e401d4-463b-45ff-91e7-7e875a962581" -append +pegasus_wf_name="gw.dax-0" -append +pegasus_wf_time="20230611T181012+0000" -append +pegasus_version="5.0.1" -append +pegasus_job_class=11 -append +pegasus_cluster_size=1 -append +pegasus_site="local" -append +pegasus_execution_sites="local" -append +pegasus_wf_xformation="pegasus::dagman" gw.dax-0.dag with environment = PATH=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/bin/intel64:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin:/software/tools/pegasus/5.0/bin/:/work/ahnitz/projects/4ogc/env/bin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/bin/intel64:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin:/software/tools/pegasus/5.0/bin/:/work/ahnitz/projects/4ogc/env/bin:/software/tools/pegasus/5.0/bin/:/work/yifan.wang/lscsoft/opt/accomlal/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games;PKG_CONFIG_PATH=/work/yifan.wang/lscsoft/opt/accomlal/lib/pkgconfig:;LAL_DATA_PATH=/atlas/recent/cbc/ROM_data/;TBBROOT=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb;TZ=:/etc/localtime;MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles;PEGASUS_PERL_DIR=/software/tools/pegasus/5.0/lib/pegasus/perl;DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/44039/bus;INTEL_LICENSE_FILE=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/licenses:/opt/intel/licenses:/work/yifan.wang/intel/licenses:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/licenses:/opt/intel/licenses:/work/yifan.wang/intel/licenses;MPM_LAUNCHER=/opt/intel/2018/debugger_2018/mpm/mic/bin/start_mpm.sh;MAIL=/var/mail/yifan.wang;LD_LIBRARY_PATH=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/mic/lib:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/ipp/lib/intel64:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.7:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.7:/opt/intel/2018/debugger_2018/iga/lib:/opt/intel/2018/debugger_2018/libipt/intel64/lib:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/mic/lib:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/ipp/lib/intel64:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.7:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.7:/opt/intel/2018/debugger_2018/iga/lib:/opt/intel/2018/debugger_2018/libipt/intel64/lib:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/work/yifan.wang/lscsoft/opt/accomlal/lib/:/work/yifan.wang/lscsoft/MultiNest/lib:;STY=2770411.pts-0.condor7;LOGNAME=yifan.wang;PWD=/work/yifan.wang/search-high-spin/inj-t11-detmass1p4/runs/O3b/1;INFOPATH=/opt/intel/2018/documentation_2018/en/debugger//gdb-ia/info/:/opt/intel/2018/documentation_2018/en/debugger//gdb-igfx/info/:/opt/intel/2018/documentation_2018/en/debugger//gdb-ia/info/:/opt/intel/2018/documentation_2018/en/debugger//gdb-igfx/info/;PYTHONPATH=/software/tools/pegasus/5.0/lib/python3.7/dist-packages/:/software/tools/pegasus/5.0/lib/python3.7/dist-packages/:/software/tools/pegasus/5.0/lib/python3.7/dist-packages/:/work/yifan.wang/eccentricity/gitlab-eccsearch/waveform/ihes-teobresum/Python:/work/yifan.wang/eccentricity/gitlab-eccsearch//waveform/PyCBC-teobresums:/work/yifan.wang/lscsoft/src/TaylorF2e:;SHELL=/bin/bash;BASH_ENV=/usr/share/modules/init/bash;INTEL_PYTHONHOME=/opt/intel/2018/debugger_2018/python/intel64/;LM_LICENSE_FILE=/opt/matlab/default/etc/license.dat;OLDPWD=/work/yifan.wang/search-high-spin;TMPDIR=/local/user/yifan.wang;LIBRARY_PATH=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/ipp/lib/intel64:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.7:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.7:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/ipp/lib/intel64:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.7:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.7:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/lib/intel64_lin:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/../tbb/lib/intel64_lin/gcc4.4;VIRTUAL_ENV=/work/ahnitz/projects/4ogc/env;PSTLROOT=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/pstl;MODULEPATH_modshare=/etc/environment-modules/modules:1:/usr/share/modules/$MODULE_VERSION/modulefiles:1:/usr/share/modules/modulefiles:1:/usr/share/modules/versions:1;LC_ALL=C;PEGASUS_PYTHON_DIR=/software/tools/pegasus/5.0/lib/python3.7/dist-packages;TERMCAP=SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual terminal:\ :DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\ :cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\ :do=^J:nd=\E[C:pt:rc=\E8:rs=\Ec:sc=\E7:st=\EH:up=\EM:\ :le=^H:bl=^G:cr=^M:it#8:ho=\E[H:nw=\EE:ta=^I:is=\E)0:\ :li#38:co#167:am:xn:xv:LP:sr=\EM:al=\E[L:AL=\E[%dL:\ :cs=\E[%i%d;%dr:dl=\E[M:DL=\E[%dM:dc=\E[P:DC=\E[%dP:\ :im=\E[4h:ei=\E[4l:mi:IC=\E[%d@:ks=\E[?1h\E=:\ :ke=\E[?1l\E>:vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l:\ :ti=\E[?1049h:te=\E[?1049l:us=\E[4m:ue=\E[24m:so=\E[3m:\ :se=\E[23m:mb=\E[5m:md=\E[1m:mh=\E[2m:mr=\E[7m:\ :me=\E[m:ms:\ :Co#8:pa#64:AF=\E[3%dm:AB=\E[4%dm:op=\E[39;49m:AX:\ :vb=\Eg:G0:as=\E(0:ae=\E(B:\ :ac=\140\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:\ :po=\E[5i:pf=\E[4i:Km=\E[M:k0=\E[10~:k1=\EOP:k2=\EOQ:\ :k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:k7=\E[18~:\ :k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:F2=\E[24~:\ :F3=\E[1;2P:F4=\E[1;2Q:F5=\E[1;2R:F6=\E[1;2S:\ :F7=\E[15;2~:F8=\E[17;2~:F9=\E[18;2~:FA=\E[19;2~:\ :FB=\E[20;2~:FC=\E[21;2~:FD=\E[23;2~:FE=\E[24;2~:kb=:\ :K2=\EOE:kB=\E[Z:kF=\E[1;2B:kR=\E[1;2A:*4=\E[3;2~:\ :*7=\E[1;2F:#2=\E[1;2H:#3=\E[2;2~:#4=\E[1;2D:%c=\E[6;2~:\ :%e=\E[5;2~:%i=\E[1;2C:kh=\E[1~:@1=\E[1~:kH=\E[4~:\ :@7=\E[4~:kN=\E[6~:kP=\E[5~:kI=\E[2~:kD=\E[3~:ku=\EOA:\ :kd=\EOB:kr=\EOC:kl=\EOD:km:;LC_CTYPE=UTF-8;CPATH=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/ipp/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mkl/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/pstl/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/ipp/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mkl/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/pstl/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/tbb/include:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/include;SHLVL=3;SLACK_BOT_TOKEN=xoxp-1889174914644-1876230893430-1893291262468-e9d8766407c79a32862f209441631a1f;MKLROOT=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mkl;CONDOR_LOCATION=/usr;LOADEDMODULES=;MANPATH=/opt/intel/2018/man/common:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/man:/opt/intel/2018/documentation_2018/en/debugger//gdb-ia/man/:/opt/intel/2018/documentation_2018/en/debugger//gdb-igfx/man/:/opt/intel/2018/man/common:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/man:/opt/intel/2018/documentation_2018/en/debugger//gdb-ia/man/:/opt/intel/2018/documentation_2018/en/debugger//gdb-igfx/man/:/work/ahnitz/projects/4ogc/env/share/man:/usr/local/man:/usr/local/share/man:/usr/share/man::;SCRATCH=/local/user/yifan.wang;I_MPI_ROOT=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi;PEGASUS_SCHEMA_DIR=/software/tools/pegasus/5.0/share/pegasus/schema;JAVA_HOME=/usr/lib/jvm/default-java;TERM=screen.xterm-256color;ENV=/usr/share/modules/init/profile.sh;LANG=en_US.UTF-8;XDG_SESSION_ID=52695;XDG_SESSION_TYPE=tty;DAALROOT=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal;IPPROOT=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/ipp;GDBSERVER_MIC=/opt/intel/2018/debugger_2018/gdb/targets/intel64/x200/bin/gdbserver;PEGASUS_PYTHON_EXTERNALS_DIR=/software/tools/pegasus/5.0/lib/pegasus/externals/python;XDG_SESSION_CLASS=user;_=/software/tools/pegasus/5.0/bin/pegasus-plan;PEGASUS_JAVA_DIR=/software/tools/pegasus/5.0/share/pegasus/java;SSH_TTY=/dev/pts/0;SSH_CLIENT=130.75.117.53 20665 22;USER=yifan.wang;CLASSPATH=/software/tools/pegasus/5.0/share/pegasus/java/accessors.jar:/software/tools/pegasus/5.0/share/pegasus/java/bcprov-jdk15on-150.jar:/software/tools/pegasus/5.0/share/pegasus/java/btf-1.2.jar:/software/tools/pegasus/5.0/share/pegasus/java/commons-lang3-3.4.jar:/software/tools/pegasus/5.0/share/pegasus/java/commons-logging.jar:/software/tools/pegasus/5.0/share/pegasus/java/commons-pool.jar:/software/tools/pegasus/5.0/share/pegasus/java/exist-optional.jar:/software/tools/pegasus/5.0/share/pegasus/java/exist.jar:/software/tools/pegasus/5.0/share/pegasus/java/gram-2.1.0.jar:/software/tools/pegasus/5.0/share/pegasus/java/gridftp-2.1.0.jar:/software/tools/pegasus/5.0/share/pegasus/java/gson-2.2.2.jar:/software/tools/pegasus/5.0/share/pegasus/java/gss-2.1.0.jar:/software/tools/pegasus/5.0/share/pegasus/java/guava-16.0.1.jar:/software/tools/pegasus/5.0/share/pegasus/java/hamcrest-core-1.3.jar:/software/tools/pegasus/5.0/share/pegasus/java/io-2.1.0.jar:/software/tools/pegasus/5.0/share/pegasus/java/jackson-annotations-2.9.10.jar:/software/tools/pegasus/5.0/share/pegasus/java/jackson-core-2.9.10.jar:/software/tools/pegasus/5.0/share/pegasus/java/jackson-coreutils-1.8.jar:/software/tools/pegasus/5.0/share/pegasus/java/jackson-databind-2.9.10.jar:/software/tools/pegasus/5.0/share/pegasus/java/jackson-dataformat-yaml-2.9.10.jar:/software/tools/pegasus/5.0/share/pegasus/java/jakarta-oro.jar:/software/tools/pegasus/5.0/share/pegasus/java/java-getopt-1.0.9.jar:/software/tools/pegasus/5.0/share/pegasus/java/javax.json-1.0.4.jar:/software/tools/pegasus/5.0/share/pegasus/java/jcodings-1.0.46.jar:/software/tools/pegasus/5.0/share/pegasus/java/joda-time-2.3.jar:/software/tools/pegasus/5.0/share/pegasus/java/joni-2.1.31.jar:/software/tools/pegasus/5.0/share/pegasus/java/json-schema-validator-1.0.41.jar:/software/tools/pegasus/5.0/share/pegasus/java/jsse-2.1.0.jar:/software/tools/pegasus/5.0/share/pegasus/java/libphonenumber-6.2.jar:/software/tools/pegasus/5.0/share/pegasus/java/log4j-1.2.17.jar:/software/tools/pegasus/5.0/share/pegasus/java/mailapi-1.4.3.jar:/software/tools/pegasus/5.0/share/pegasus/java/msg-simple-1.1.jar:/software/tools/pegasus/5.0/share/pegasus/java/myproxy-2.1.0.jar:/software/tools/pegasus/5.0/share/pegasus/java/mysql-connector-java-5.1.47.jar:/software/tools/pegasus/5.0/share/pegasus/java/pegasus-aws-batch.jar:/software/tools/pegasus/5.0/share/pegasus/java/pegasus.jar:/software/tools/pegasus/5.0/share/pegasus/java/postgresql-8.1dev-400.jdbc3.jar:/software/tools/pegasus/5.0/share/pegasus/java/resolver.jar:/software/tools/pegasus/5.0/share/pegasus/java/rhino-1.7R4.jar:/software/tools/pegasus/5.0/share/pegasus/java/snakeyaml-1.25.jar:/software/tools/pegasus/5.0/share/pegasus/java/sqlite-jdbc-3.8.11.2.jar:/software/tools/pegasus/5.0/share/pegasus/java/ssl-proxies-2.1.0-patched.jar:/software/tools/pegasus/5.0/share/pegasus/java/super-csv-2.4.0.jar:/software/tools/pegasus/5.0/share/pegasus/java/uri-template-0.9.jar:/software/tools/pegasus/5.0/share/pegasus/java/vdl.jar:/software/tools/pegasus/5.0/share/pegasus/java/xercesImpl.jar:/software/tools/pegasus/5.0/share/pegasus/java/xml-apis-1.4.01.jar:/software/tools/pegasus/5.0/share/pegasus/java/xmlParserAPIs.jar:/software/tools/pegasus/5.0/share/pegasus/java/xmldb.jar:/software/tools/pegasus/5.0/share/pegasus/java/xmlrpc.jar:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/mpi.jar:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/lib/daal.jar:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/mpi.jar:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/lib/daal.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/apache-client-2.0.0-preview-2.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/batch-2.0.0-preview-2.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/commons-logging-api-1.1.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/core-2.0.0-preview-2.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/http-client-spi-2.0.0-preview-2.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/httpclient-4.5.2.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/httpcore-4.4.4.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/jackson-annotations-2.8.8.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/jackson-databind-2.8.8.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/jackson-jr-objects-2.9.0.pr4.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/joda-time-2.8.1.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/jopt-simple-5.0.4.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/logs-2.0.0-preview-2.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/metrics-spi-2.0.0-preview-2.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/netty-nio-client-2.0.0-preview-2.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/s3-2.0.0-preview-2.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/slf4j-api-1.7.25.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/slf4j-log4j12-1.7.25.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/thirdparty-logging.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/thirdparty.jar:/software/tools/pegasus/5.0/share/pegasus/java/aws/utils-2.0.0-preview-2.jar;C_INCLUDE_PATH=/work/yifan.wang/lscsoft/opt/accomlal/include:;WINDOW=0;PEGASUS_BIN_DIR=/software/tools/pegasus/5.0/bin;SSH_CONNECTION=130.75.117.53 20665 130.75.116.24 22;MODULESHOME=/usr/share/modules;PEGASUS_CONF_DIR=/software/tools/pegasus/5.0/etc;TMP=/local/user/yifan.wang;PYCBC_WAVEFORM=taylorf2e;NLSPATH=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64_lin/locale/%l_%t/%N:/opt/intel/2018/debugger_2018/gdb/intel64/share/locale/%l_%t/%N:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64_lin/locale/%l_%t/%N:/opt/intel/2018/debugger_2018/gdb/intel64/share/locale/%l_%t/%N;PEGASUS_ORIG_CLASSPATH=/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/mpi.jar:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/lib/daal.jar:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/mpi.jar:/opt/intel/2018/compilers_and_libraries_2018.0.128/linux/daal/lib/daal.jar;MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl;LIGO_DATAFIND_SERVER=datafind.atlas.local:80;GW_DATAFIND_SERVER=datafind.atlas.local:80;XDG_RUNTIME_DIR=/run/user/44039;GDB_CROSS=/opt/intel/2018/debugger_2018/gdb/intel64/bin/gdb-ia;PEGASUS_SHARE_DIR=/software/tools/pegasus/5.0/share/pegasus;HOME=/work/yifan.wang; 2023.06.11 18:10:22.239 GMT: 2023.06.11 18:10:22.245 GMT: ----------------------------------------------------------------------- 2023.06.11 18:10:22.250 GMT: File for submitting this DAG to HTCondor : gw.dax-0.dag.condor.sub 2023.06.11 18:10:22.255 GMT: Log of DAGMan debugging messages : gw.dax-0.dag.dagman.out 2023.06.11 18:10:22.260 GMT: Log of HTCondor library output : gw.dax-0.dag.lib.out 2023.06.11 18:10:22.265 GMT: Log of HTCondor library error messages : gw.dax-0.dag.lib.err 2023.06.11 18:10:22.270 GMT: Log of the life of condor_dagman itself : gw.dax-0.dag.dagman.log 2023.06.11 18:10:22.275 GMT: 2023.06.11 18:10:22.280 GMT: -no_submit given, not submitting DAG to HTCondor. You can do this with: 2023.06.11 18:10:22.290 GMT: ----------------------------------------------------------------------- 2023.06.11 18:10:22.296 GMT: [DEBUG] condor_submit_dag exited with status 0 2023.06.11 18:10:22.300 GMT: [DEBUG] Updated environment for dagman is environment = _CONDOR_SCHEDD_ADDRESS_FILE=/local/condor/spool/.schedd_address;_CONDOR_MAX_DAGMAN_LOG=0;_CONDOR_SCHEDD_DAEMON_AD_FILE=/local/condor/spool/.schedd_classad;_CONDOR_DAGMAN_LOG=gw.dax-0.dag.dagman.out;PEGASUS_METRICS=true; 2023.06.11 18:10:22.301 GMT: [INFO] event.pegasus.code.generation dax.id gw.dax-0 (8.997 seconds) - FINISHED 2023.06.11 18:10:22.302 GMT: [DEBUG] Executing /software/tools/pegasus/5.0/bin/pegasus-db-admin update -t master -c /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/pegasus.16970775812990819111.properties 2023.06.11 18:10:23.938 GMT: Database version: '5.0.1' (sqlite:////work/yifan.wang/.pegasus/workflow.db) 2023.06.11 18:10:23.989 GMT: [DEBUG] pegasus-db-admin exited with status 0 2023.06.11 18:10:23.990 GMT: [DEBUG] Executing /software/tools/pegasus/5.0/bin/pegasus-db-admin create -Dpegasus.catalog.replica.db.url=jdbc:sqlite:/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replicas.db -Dpegasus.catalog.replica=JDBCRC -Dpegasus.catalog.replica.db.driver=sqlite -t jdbcrc -c /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/pegasus.16970775812990819111.properties 2023.06.11 18:10:26.605 GMT: Pegasus database was successfully created. 2023.06.11 18:10:26.610 GMT: Database version: '5.0.1' (sqlite:////local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replicas.db) 2023.06.11 18:10:26.657 GMT: [DEBUG] pegasus-db-admin exited with status 0 2023.06.11 18:10:26.657 GMT: Output replica catalog set to jdbc:sqlite:/local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work/gw.dax-0.replicas.db 2023.06.11 18:10:26.658 GMT: [DEBUG] Executing /software/tools/pegasus/5.0/bin/pegasus-run --nogrid /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work 2023.06.11 18:10:27.205 GMT: Submitting to condor gw.dax-0.dag.condor.sub 2023.06.11 18:10:27.258 GMT: 2023.06.11 18:10:27.263 GMT: Your workflow has been started and is running in the base directory: 2023.06.11 18:10:27.268 GMT: 2023.06.11 18:10:27.273 GMT: /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work 2023.06.11 18:10:27.278 GMT: 2023.06.11 18:10:27.284 GMT: *** To monitor the workflow you can run *** 2023.06.11 18:10:27.289 GMT: 2023.06.11 18:10:27.294 GMT: pegasus-status -l /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work 2023.06.11 18:10:27.299 GMT: 2023.06.11 18:10:27.304 GMT: *** To remove your workflow run *** 2023.06.11 18:10:27.309 GMT: 2023.06.11 18:10:27.314 GMT: pegasus-remove /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work 2023.06.11 18:10:27.319 GMT: [DEBUG] Submission of workflow exited with status 0 2023.06.11 18:10:27.323 GMT: [DEBUG] Sending Planner Metrics to [1 of 1] http://metrics.pegasus.isi.edu/metrics 2023.06.11 18:10:28.634 GMT: [DEBUG] Metrics succesfully sent to the server 2023.06.11 18:10:28.635 GMT: Time taken to execute is 14.989 seconds 2023.06.11 18:10:28.635 GMT: [INFO] event.pegasus.planner planner.version 5.0.1 (16.167 seconds) - FINISHED Querying Pegasus database for workflow stored in /local/user/yifan.wang/pycbc-tmp.x6CoQQZOfW/work This may take up to 120 seconds. Please wait......... Done. Workflow submission completed successfully. The Pegasus dashboard URL for this workflow is: https://condor7.atlas.local/pegasus/u/yifan.wang/r/501/w?wf_uuid=85e401d4-463b-45ff-91e7-7e875a962581 Note that it make take a while for the dashboard entry to appear while the workflow is parsed by the dashboard. The delay can be on the order of one hour for very large workflows.