One obvious fix is to avoid copying over files that are still being written. We can now easily tell which ones those are, at least for the regular xml/sql dump runs. This patch implements it for that case: https://gerrit.wikimedia.org/r/#/c/385203/

I did some research earlier and looked at the rsync code; versions 3.0.0 and greater create the file list incrementally, which uses much less memory than the older versions. Anything running precise and up will have 3.0.0, so we can practically rule out use of older versions by our mirrors. I tried checking the filecount lookahead and some other details, but tl;dr is that I still wonder if doing smaller subdirs at a time would be less resource-intensive. Needs some testing.

Setup time before file list transmission seems nearly the same for top level directories and subdirs, tested on dataset1001 which has a very large filesystem. Things yet to be tested; making the include/exclude list less complex or shorter; using separate rsync stanzas for subdirectories to see if that's faster.

We're in pretty good shape now, rsycning only complete files, as soon as they are produced, and with less populated filesystems on the dumpsdata server side. Subdirectory rsyncing didn't help any. Steps forward should now be coordinated with @Bstorm to see what might be done on the labstore side of these rsyncs.