File Transfer Best Practices
For transfers between file systems within the data center and small data transfers (<100GB) outside of the NREL network, scp and rsync are the recommended tools. For large data transfers between NREL and another data center, or even a laptop off-site, Globus is the recommended transfer tool. You'll find more details about each transfer method in the next few sections.
Transferring Data between filesystems on the NREL network
rsync is the recommended tool for transferring data between NREL systems. It allows you to easily restart transfers if they fail, and also provides more consistency when dealing with symbolic links, hard links, and sparse files than either scp or cp. It is recommended you do not use compression for transfers within NREL systems. An example command is:
rsync -aP /scratch/username/dataset1/ /mss/users/username/dataset1/
For Small Transfers (<100GB) outside of the NREL Network, use rsync or scp
rsync or scp will be your best option for small transfers (<100GB) outside of the NREL network. If your rsync/scp transfers are taking hours to complete then you should consider using Globus.
If you're transferring many files then you should use rsync:
$ rsync -azP /mss/users/username/dataset1/ user@desthost:/home/username/dataset1/
If you're transferring an individual file then use scp:
$ scp /home/username/example.tar.gz user@desthost:/home/username/
Mass Storage has quotas that limit the number of individual files you can store. If you are copying thousands of files then it is best to tar these files up prior to copying to Mass Storage.
From Peregrine you can use wget or curl to transfer datasets less than 100GB. You should avoid using wget or curl to transfer datasets over 100GB since web traffic goes through the proxy and is not optimized for large data transfers. If your only option for downloading large datasets (>100GB) involves wget or curl, please contact email@example.com and request assistance.
For Large transfers (>100GB) outside of the NREL Network, use Globus
Globus is optimized for file transfers between data centers and anything outside of the NREL network. It will be several times faster than any other tools you will have available. Documentation about Globus is available on the Globus Services page.
Transferring Data to and from a Windows System or Laptop
WinSCP is the recommended tool for transferring files to and from NREL Linux systems using Windows. You can find documentation on using WinSCP on the WinSCP page. If your Windows System or Laptop is outside of the NREL network then Globus will provide better performance than WinSCP. Documentation about Globus is available on the Globus Services page.