| ArielGlenn | |
| Jan 6 2017, 7:40 PM |
| F5303966: labstore1003_load.png | |
| Jan 18 2017, 6:13 AM |
| F5303967: labstore1003_network.png | |
| Jan 18 2017, 6:13 AM |
As reported by Bryan W:
The last two two cycles, the dumps have not been moved to Labs as the
individual files complete.
On the 20171220 cycle, the files were not transferred until after the
20170101 cycle had started.
The 20170101 files are not showing up on Labs.
Woops yes, thank you. Also a note that Bryan has told me via email that the dumps started transferring. Nonetheless I should look into the cause for the delay until then.
A new run will be starting on the first of the month, so I'll watch the transfer situation. A reminder that the list of files to transfer is generated by cron from a snapshot host once a day, and the rsync itself runs on a different host once a day, so there may be a wait based on that.
Yes there is a delay! And I even know why, and an appropriate fix. Testing a change to the cron script right now.
The long version of this is that the files of dumps that are 'in progress' were not included in the files to be rsynced, after refactoring of the script that generates the file lists. we do have a list that includes these directories and/or files explicitly, with appropriate include/exclude arguments to rsync. So the rsync bash script can be rewritten to use this, invoking rsync directly instead of a hacky python script.
That rewrite is what I'm testing now, verifying that the list of files it expects to rsync are the ones we want.
Change 336204 had a related patch set uploaded (by ArielGlenn):
dataset1001 rsync to labs of dumps can now use explicit inclusion list
Change 336204 merged by ArielGlenn:
[operations/puppet@production] dataset1001 rsync to labs of dumps can now use explicit inclusion list
Change 399261 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] add dumpsgen user to labstore1003 for dumps cron cleanup job
Change 399261 merged by ArielGlenn:
[operations/puppet@production] add dumpsgen user to labstore1003 for dumps cron cleanup job
I checked the files copied, and they are up to date as of the cron job run, for any completed files. We now copy any completed files over without waiting for the dump run to complete and be known to be good. Caveat emptor! But it does mean that this task can be closed.
