Jaime already started to work on a framework to transfer files (in the DB context: to clone databases):
https://gerrit.wikimedia.org/r/#/c/326155/
The transfer method must: * Be as fast as the network bandwidth allows it, with configurable throttling * Be easy to use (take automatic decisions when safe) * Allow for both single files and entire directories to be synced. The directories can be of thousands of small files * Keep the original permissions and ownership * Have in-place checks to avoid doing damaging stuff * Allow encryption * Allow configurable compression * Allow configurable resource taking (e.g. number of CPUs) * Checksum contents before and after the copy to check it has been done successfuly * Allow multicast-like transfers from 1 server to many * Report the status at any time, and if it fails, why * Handle the firewall automatically * Not require a constantly open port or service The current code is just the barebones, it has to be integrated with volan's packages for remote code execution. Because the above, rsync is not enough. We have to give a look to multi-thread FTP, tar + socat with user encryption, and bittorent.