Fastest way to move multiple terabytes of information

John Doe asked:

I need to move multiple terabytes of information from one NFS system (The NFS system is in AWS) to amazon EFS, but I need it to move that that data the fastest way possible. I’ve tried rsync, but it takes too long, and I’ve also tried using parallel with rsync but it does not give me the results I need.

The data consists in multiple directories with many small files

Are there any tried and tested ways to move data very quickly while still making sure that it’s not corrupted when it arrives (Like rsync)?

My answer:


The fastest way to move such large volumes of data is still Sneakernet. Thus, Amazon Snowball. This device gets shipped to your location, where it can move up to 80 terabytes of data directly off your local network via a 10 Gigabit Ethernet connection. It then gets shipped back to Amazon where they upload the data into your Amazon account.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.