By "clean up" here I mean:
This includes methods for selecting prefetch files from previous runs and stub files for the current run.
Change 575581 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] cleanup of page content dumps run()
Change 575584 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] fix up file list methods
Change 575585 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] convert all file list methods to use common args
Change 575586 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] move StubProvider out to its own module
Change 575587 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] move some dfname/pagerange munging methods to their own class
Change 575588 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] move some output file listing methods to their own module
Change 575589 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] use only jobFileLister instance methods in other modules
Change 575591 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] add some unit tests for prefetch arg generation
Because I have already thoroughly tested these patches, and the window for deploys is basically today and tomorrow, I'll merge and deploy tomorrow so that they are in time for the March 1 run. This will let me stack up another set of commits for the March 20th run rather than bundling these with those all together then.
Change 575581 merged by ArielGlenn:
[operations/dumps@master] cleanup of page content dumps run()
Change 575584 merged by ArielGlenn:
[operations/dumps@master] fix up file list methods
Change 575585 merged by ArielGlenn:
[operations/dumps@master] convert all file list methods to use common args
Change 575586 merged by ArielGlenn:
[operations/dumps@master] move StubProvider out to its own module
Change 575587 merged by ArielGlenn:
[operations/dumps@master] move some dfname/pagerange munging methods to their own class
Change 575588 merged by ArielGlenn:
[operations/dumps@master] move some output file listing methods to their own module
Change 575589 merged by ArielGlenn:
[operations/dumps@master] use only jobFileLister instance methods in other modules
Change 575591 merged by ArielGlenn:
[operations/dumps@master] add some unit tests for prefetch arg generation
Mentioned in SAL (#wikimedia-operations) [2020-03-01T06:02:48Z] <ariel@deploy1001> Started deploy [dumps/dumps@8376c62]: refactor page content jobs, prefetch, and output file listings: see T246465
Mentioned in SAL (#wikimedia-operations) [2020-03-01T06:02:53Z] <ariel@deploy1001> Finished deploy [dumps/dumps@8376c62]: refactor page content jobs, prefetch, and output file listings: see T246465 (duration: 00m 04s)
Change 577225 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] rename 'parts' attribute of Dump subclasses to something more accurate
Change 577226 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] make value of 'parts' in the file listing methods be None or a list
Change 577228 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] New class for output file listing methods to move them out of jobs code
Change 578477 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] clean up file list method docs, tighten up code
These last patches are ready to go once the wikidata dumps run finishes up. After that this ticket can be closed.
Change 577225 merged by ArielGlenn:
[operations/dumps@master] rename 'parts' attribute of Dump subclasses to something more accurate
Change 577226 merged by ArielGlenn:
[operations/dumps@master] make value of 'parts' in the file listing methods be None or a list
Change 577228 merged by ArielGlenn:
[operations/dumps@master] New class for output file listing methods to move them out of jobs code
Change 578477 merged by ArielGlenn:
[operations/dumps@master] clean up file list method docs, tighten up code
The new run has started with these patches deployed. If there are any issues during the run they can be discussed in a new ticket.