Right now we keep only a couple of full dump runs on the dumps generation nfs servers. Ideally we could keep just one besides the current run, and keep prefetch files for 4 runs or even more (7z for the history files, bz2 for everything else) in some separate directory which could be checked. This is in case the prefetch file from a previous run turns out to have problems.
A side effect is that this would allow us to have less space on the generation nfs servers.
There are a couple of very drafty changesets on this already, see https://gerrit.wikimedia.org/r/#/c/423241/ and https://gerrit.wikimedia.org/r/#/c/423242/ The approach outlined in them may change completely.
I'm getting ready to abandon those above patchsets; what we really want is a nice way to keep just the prefetch files from some old dumps (plus the status file), instead of having to have a separate directory and copy/move them in and go look them up there etc.
Change 432135 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] ad ability to keep some dumps files and remove others, during cleanup
Change 432135 merged by ArielGlenn:
[operations/puppet@production] add ability to keep some dumps files and remove others, during cleanup
Change 432378 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] keep dump prefetch files longer on dumps generation nfs servers
Change 432378 merged by ArielGlenn:
[operations/puppet@production] keep dump prefetch files longer on dumps generation nfs servers
Change 434144 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] keep partial dumps on the dump generation nfs servers, not the web servers
Change 434144 merged by ArielGlenn:
[operations/puppet@production] keep partial dumps on the dump generation nfs servers, not the web servers
Keeping this open until tomorrow morning, when the first job that should preserve partial dumps will run.
Change 434468 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] keep around dump prefetch files from small wikis too
Change 434468 merged by ArielGlenn:
[operations/puppet@production] keep around dump prefetch files from small wikis too
I need to wait one more run to make sure that the cleanup job runs properly, and then this can be closed.