How to use ligodev to move data off Atlas
How to use ligodev to move data off Atlas
Based on the guideline on large transfers, here is a brief checklist, how to prepare a transfer efficiently:
- Log into
ligodev.atlas.aei.uni-hannover.de
with gsissh. - You will arrive in
/home/$USER
which is read-only for the time being. - Start a shell under
screen
ortmux
to be able to log out when running commands taking a long time to complete. - Choose a directory (using
/path/xfer/
here-after) you want to prepare for transfer and check with bothdu -sh --apparent-size /path/xfer
anddu -sh /path/xfer
how large the directory is - if both numbers differ wildly, please sent us the path to the directory by email, as a number of files may need to be copied from tape to disk. - Use tar/pigz to compress the directory and store it on the local
disks of
ligodev
, i.e.tar c /path/xfer | pigz --processes 4 --best --stdout > /local/user/$(whoami)/xfer.tar.gz
. - When done, transfer the tarball via globus-url-copy to remote site,
e.g.
globus-url-copy -p 16 -tcp-bs $((2**20)) -vb file:///local/user/$(whoami)/xfer.tar.gz gsiftp://REMOTE/home/$USER/xfer.tar.gz
. - Optionally, check if local and remote files have the same hecksum,
e.g. use
md5sum
,sha1sum
, … on both and compare the results. - IMPORTANT: When the transfer is done, you must delete the local
file
/local/user/$(whoami)/xfer.tar.gz
again to make room for other peoples’ transfers! - Repeat the process for any directory structure you want to transfer.
(Note below: The reason why /home/$USER
is currently mounted
read-only is that many people started to create the tarballs in their
home directories which in turn overloaded the HSM and caused the
underlying DDN disk array to collapse multiple times.)