A recommended storage pattern is to have the master copy of data on Atlas (project folder) and to only store data on the HPC Clusters temporarily for the duration of computation/analysis. All analysis performed on the HPC Clusters must therefore be regularly transferred to Atlas. This How-to Card describes the different methods to transfer data between Atlas and the UL HPC Clusters. The three recommended methods to transfer data are:
1.[Via laptop with ```scp``` or ```rsync```]()
2.[Via dedicated Virtual Machine (VM)]()
3.[Via Large File Transfer (LFT)]()
## 1. Via laptop using scp or rsync
The most common commands for data transfers over SSH are ```scp``` and ```rsync```.
*```scp```: transfers all files and directories.
*```rsync```: transfers only the files which differs between the source and the destination.
While both commands ensure a secure transfer of data between the UL HPC Clusters and Atlas, ```rsync``` is in general preferred over ```scp``` due to it's selective incremental transfer of files, which makes it faster than ```scp```.
Before transferring the data over SSH, you must mount Atlas to your local laptop. Depending on your operating system, you can find a guide on the [Service portal](https://service.uni.lu/sp?id=kb_article_view&sysparm_article=KB0010233).
Finally, to use use ```rsync``` and ```scp``` for data transfer, please visit the [UL HPC documentation](https://hpc-docs.uni.lu/data/transfer/#data-transfer-tofromwithin-ul-hpc-clusters).
## 2. Via dedicated Virtual Machine (VM) using rsync
Data can be transferred via a dedicated VM, which can be requested through [ServiceNow](https://service.uni.lu/sp?id=sc_cat_item&table=sc_cat_item&sys_id=49956812db3fa010ca53454039961978).
Instead of transferring data between Atlas and UL HPC Clusters through the laptop as described above, the transfer will go through the dedicated VM. Once connected to the VM and mounted to Atlas, the ```rsync``` command can be use in similar way as described in the [UL HPC documentation]((https://hpc-docs.uni.lu/data/transfer/#data-transfer-tofromwithin-ul-hpc-clusters)). This method is recommended for **recurrent transfer of big data**.
> Note: For larger transfers between Atlas and UL HPC Clusters, you may want to run some operations in the background using ```screen``` or ```tmux```.
## 3. Via Large File Transfer (LFT)
An alternative solution for transferring data between Atlas and UL HPC Clusters is to use LFT, which can transfer high data volumes of several tera bytes. However, LFT is only an advantage to use, if data is already on LFT (e.g. received by collaborators). In this case, data can easily be uploaded to Atlas manually, ensuring a master copy of the raw data. A copy of the data can then directly be uploaded to UL HPC Clusters for computational analysis.
Please visit the dedicated [How-to Card on LFT]({{'/?exchange-channels:lft' | relative_url }}) to see further information.
> NOTE: LFT is only recommended for the use case mentioned above, where raw data is already received via LFT, as this method requires installation of dedicated software and consists of lengthy steps in comparison to the first two methods.
