Page MenuHomePhabricator
Create Task
Maniphest T179942

Move misc dump cron jobs from dataset1001 to dumpsgen1001
Closed, ResolvedPublic
Actions

Description

Xml/sql dumps run on the dumpsdata hosts, move the misc dump cron jobs there too.

  • move all references to user names out of the snapshot and dumps modules to profiles
  • move remaining references to /mnt/data or /mnt/dumpsdata in snapshot modules out to profiles
  • move remaining references to dataset1001 or dumpsdata1001 or ms1001 in modules out to profiles
  • move cron jobs one at a time:
      • stop job
      • change config file reference and read/write path in manifests
      • chown files to right user on dataset1001, change user in the cron job manifest
      • add to rolling rsyncs to dumpsdata1002, dataset1001
      • re-enable job
      • add cleanup stanza to daiy cron job on dataset1001 that currently just cleans up old xml/sql dumps
    • wikidata jobs
    • cirrussearch
    • categoriesrdf
    • dump_global_blocks
    • contentxlation
    • addschanges
    • mediaperprojectlists
    • pagetitles

Details

Show related patches Customize query in gerrit

Event Timeline

ArielGlenn created this task.Nov 7 2017, 2:51 PM
ArielGlenn moved this task from Backlog to Active on the Dumps-Generation board.Nov 7 2017, 3:00 PM
gerritbot subscribed.Nov 7 2017, 3:01 PM

Change 389719 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] move explicit references to users and groups out of snapshot module

https://gerrit.wikimedia.org/r/389719

gerritbot added a project: Patch-For-Review.Nov 7 2017, 3:01 PM
gerritbot added a comment.Nov 7 2017, 3:47 PM

Change 389719 merged by ArielGlenn:
[operations/puppet@production] move explicit references to users and groups out of snapshot module

https://gerrit.wikimedia.org/r/389719

ArielGlenn added a comment.Nov 7 2017, 4:19 PM

https://gerrit.wikimedia.org/r/#/c/389728/ deals with the last named user in the snapshot manifests.

gerritbot added a comment.Nov 7 2017, 5:25 PM

Change 389745 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] move references to datasets use from dumps module out to profile

https://gerrit.wikimedia.org/r/389745

gerritbot added a comment.Nov 8 2017, 1:31 PM

Change 389745 merged by ArielGlenn:
[operations/puppet@production] move references to datasets use from dumps module out to profile

https://gerrit.wikimedia.org/r/389745

ArielGlenn added a comment.Nov 8 2017, 2:34 PM
This comment was removed by ArielGlenn.
gerritbot added a comment.Nov 8 2017, 3:36 PM

Change 390000 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] move hardcoded references to dump server mount points out to profile

https://gerrit.wikimedia.org/r/390000

gerritbot added a comment.Nov 8 2017, 5:03 PM

Change 390000 merged by ArielGlenn:
[operations/puppet@production] move hardcoded references to dump server mount points out to profile

https://gerrit.wikimedia.org/r/390000

gerritbot added a comment.Nov 8 2017, 8:53 PM

Change 390085 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] move last hardcoded user names out of snapshot modules to profiles

https://gerrit.wikimedia.org/r/390085

gerritbot added a comment.Nov 8 2017, 9:12 PM

Change 390085 merged by ArielGlenn:
[operations/puppet@production] move last hardcoded user names out of snapshot modules to profiles

https://gerrit.wikimedia.org/r/390085

ArielGlenn updated the task description. (Show Details)Nov 8 2017, 9:20 PM
gerritbot added a comment.Nov 8 2017, 10:29 PM

Change 390165 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] clean up dumps web server rsync to its fallback

https://gerrit.wikimedia.org/r/390165

gerritbot added a comment.Nov 9 2017, 1:21 PM

Change 390165 merged by ArielGlenn:
[operations/puppet@production] clean up dumps web server rsync to its fallback

https://gerrit.wikimedia.org/r/390165

ArielGlenn updated the task description. (Show Details)Nov 9 2017, 1:30 PM
gerritbot added a comment.Nov 17 2017, 5:10 PM

Change 392070 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] Prep for moving misc dump cron jobs to dumpsdata host

https://gerrit.wikimedia.org/r/392070

gerritbot added a comment.Nov 17 2017, 5:37 PM

Change 392070 merged by ArielGlenn:
[operations/puppet@production] Prep for moving misc dump cron jobs to dumpsdata host

https://gerrit.wikimedia.org/r/392070

gerritbot added a comment.Nov 17 2017, 6:39 PM

Change 392085 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] set up directory for misc cron dumps on dumpsdata, move first cron job

https://gerrit.wikimedia.org/r/392085

gerritbot added a comment.Nov 17 2017, 7:35 PM

Change 392085 merged by ArielGlenn:
[operations/puppet@production] set up directory for misc cron dumps on dumpsdata, move first cron job

https://gerrit.wikimedia.org/r/392085

gerritbot added a comment.Nov 17 2017, 8:22 PM

Change 392099 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] cron jobs on the dumpsdatahosts use the new dumpsgen user

https://gerrit.wikimedia.org/r/392099

gerritbot added a comment.Nov 17 2017, 8:25 PM

Change 392099 merged by ArielGlenn:
[operations/puppet@production] cron jobs on the dumpsdatahosts use the new dumpsgen user

https://gerrit.wikimedia.org/r/392099

gerritbot added a comment.Nov 21 2017, 11:19 AM

Change 392625 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] rsync misc dumps (everything but xml/sql) to fallback hosts, labstore1006

https://gerrit.wikimedia.org/r/392625

gerritbot added a comment.Nov 22 2017, 1:16 PM

Change 392625 merged by ArielGlenn:
[operations/puppet@production] rsync misc dumps (everything but xml/sql) to fallback hosts, labstore1006

https://gerrit.wikimedia.org/r/392625

gerritbot added a comment.Nov 24 2017, 3:15 PM

Change 393245 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] clean up old misc dump output files from cron jobs on dump hosts

https://gerrit.wikimedia.org/r/393245

gerritbot added a comment.Dec 2 2017, 5:10 PM

Change 393245 merged by ArielGlenn:
[operations/puppet@production] clean up old misc dump output files from cron jobs on dump hosts

https://gerrit.wikimedia.org/r/393245

gerritbot added a comment.Dec 3 2017, 3:26 PM

Change 394852 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] move various misc dump cron jobs to new nfs server

https://gerrit.wikimedia.org/r/394852

gerritbot added a comment.Dec 4 2017, 12:16 PM

Change 394852 merged by ArielGlenn:
[operations/puppet@production] move various misc dump cron jobs to new nfs server

https://gerrit.wikimedia.org/r/394852

gerritbot added a comment.Dec 5 2017, 8:36 PM

Change 395612 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] move adds-changes (so-called incrementals) dump cron to dumpsdata hosts

https://gerrit.wikimedia.org/r/395612

gerritbot added a comment.Dec 6 2017, 4:01 PM

Change 395612 merged by ArielGlenn:
[operations/puppet@production] move adds-changes (so-called incrementals) dump cron to dumpsdata hosts

https://gerrit.wikimedia.org/r/395612

ArielGlenn added subscribers: Nikerabbit, hoo.Dec 7 2017, 10:34 AM

We're down to just wikidata and contenttranslation now. Once I have the thumbs up from @hoo, I'll do wikidata (in time for next week?) I'll check with @Nikerabbit about when he thinks the contenttranslation dumps are stable again before moving those (maybe also next week?)

gerritbot added a comment.Dec 8 2017, 10:44 PM

Change 396541 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] move content translation dumps to new nfs server

https://gerrit.wikimedia.org/r/396541

gerritbot added a comment.Dec 9 2017, 9:19 AM

Change 396541 merged by ArielGlenn:
[operations/puppet@production] move content translation dumps to new nfs server

https://gerrit.wikimedia.org/r/396541

gerritbot added a comment.Dec 9 2017, 12:17 PM

Change 396574 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] move wikidata weekly dumps to new nfs server

https://gerrit.wikimedia.org/r/396574

gerritbot added a comment.Dec 10 2017, 10:48 AM

Change 396574 merged by ArielGlenn:
[operations/puppet@production] move wikidata weekly dumps to new nfs server

https://gerrit.wikimedia.org/r/396574

ArielGlenn added a comment.Dec 10 2017, 3:20 PM

The last change went in today, for wikidatawiki weekly crons. Keeping this open for one week to make sure everything looks good.

ArielGlenn moved this task from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.Dec 10 2017, 3:20 PM
ArielGlenn updated the task description. (Show Details)Dec 16 2017, 10:58 AM
ArielGlenn updated the task description. (Show Details)
ArielGlenn closed this task as Resolved.Dec 16 2017, 11:02 AM

These all have completed at least one clean run. Time to close the ticket :-)

ArielGlenn moved this task from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.Dec 16 2017, 11:03 AM
ArielGlenn moved this task from Short-term backlog to Done on the User-ArielGlenn board.Aug 7 2018, 10:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL