Page MenuHomePhabricator
Create Task
Maniphest T133694

Move misc cron jobs from snapshot1003 to new server(s)
Closed, ResolvedPublic
Actions

Description

Dumps run on the new snapshot hosts (1005-1007) but the misc cron jobs do not yet. They need clean up, in some cases rewrite to use the same base script instead of several different scripts, and the puppet classes need cleanup as well. Then a role for the misc cron jobs could be applied to the new servers.

Waiting for verification that cron runs properly:

  • media list generation per project
  • page title list generation
  • cirrus search dumps
  • central auth dumps
  • generation of lists of last n good dumps
  • wikidata dumps - runs Mon

Details

SubjectRepoBranchLines +/-
move wikidata json, ttl dumps cron job to snapshot1007operations/puppetproduction+6 -45
move generation of lists of good dumps to snapshot1007operations/puppetproduction+2 -17
move cron job for central auth dump to snapshot1007operations/puppetproduction+6 -19
move cron job for pagetitle list generation to snapshot1007operations/puppetproduction+7 -27
enable generation of media lists per project on snapshot1007operations/puppetproduction+25 -1
cleanup media list generation cron job, remove from snapshot1003operations/puppetproduction+24 -63
Customize query in gerrit

Related Objects
Search...

Event Timeline

ArielGlenn triaged this task as Medium priority.Apr 26 2016, 2:18 PM
ArielGlenn created this task.
ArielGlenn added a parent task: T131113: deploy new snapshot hosts in eqiad.
ArielGlenn moved this task from Backlog to Active on the Dumps-Generation board.
• MZMcBride subscribed.Apr 26 2016, 2:40 PM
ArielGlenn moved this task from Active to Up Next on the Dumps-Generation board.Apr 26 2016, 3:19 PM
ArielGlenn mentioned this in T140439: reinstall snapshot1001.eqiad.wmnet with RAID.Jul 15 2016, 6:38 AM
ArielGlenn added a parent task: T140439: reinstall snapshot1001.eqiad.wmnet with RAID.Jul 21 2016, 8:47 AM
ArielGlenn added a comment.EditedJul 25 2016, 10:43 AM

https://gerrit.wikimedia.org/r/#/c/300437/ fixes up onallwikis script so it can be used for more of these misc cronjobs

ArielGlenn mentioned this in rOPUP49a6fb6dc343: dumps misc crons: convert media list generation to use onallwikis.Jul 25 2016, 10:50 AM
ArielGlenn mentioned this in rOPUPc44eea68cc27: cleanup media list generation cron job, remove from snapshot1003.Jul 25 2016, 11:30 AM
ArielGlenn updated the task description. (Show Details)Jul 25 2016, 11:32 AM
ArielGlenn mentioned this in rOPUP91a549386eb6: enable generation of media lists per project on snapshot1007.Jul 25 2016, 12:11 PM
ArielGlenn moved this task from Up Next to Active on the Dumps-Generation board.Jul 25 2016, 12:30 PM
ArielGlenn updated the task description. (Show Details)Jul 25 2016, 12:33 PM
ArielGlenn updated the task description. (Show Details)
ArielGlenn mentioned this in rOPUP4036b72ba3bd: move cron job for pagetitle list generation to snapshot1007.Jul 25 2016, 12:48 PM
ArielGlenn updated the task description. (Show Details)Jul 25 2016, 12:57 PM
ArielGlenn added a comment.Jul 25 2016, 1:23 PM

https://gerrit.wikimedia.org/r/#/c/300865/ merged to move cirrusdumps to snapshot1007.

ArielGlenn updated the task description. (Show Details)Jul 25 2016, 1:24 PM
ArielGlenn mentioned this in rOPUP13a088486242: move cron job for central auth dump to snapshot1007.Jul 25 2016, 2:59 PM
ArielGlenn mentioned this in rOPUP4e79beacd175: move cron job for central auth dump to snapshot1007.Jul 25 2016, 3:03 PM
ArielGlenn mentioned this in rOPUP0d3bd5eee1d6: move cron job for central auth dump to snapshot1007.Jul 25 2016, 3:17 PM
gerritbot subscribed.Jul 25 2016, 3:19 PM

Change 300878 merged by ArielGlenn:
move cron job for central auth dump to snapshot1007

https://gerrit.wikimedia.org/r/300878

ArielGlenn mentioned this in rOPUPa982a8685526: move cron job for central auth dump to snapshot1007.Jul 25 2016, 3:22 PM
ArielGlenn updated the task description. (Show Details)Jul 25 2016, 3:26 PM
gerritbot added a comment.Jul 25 2016, 3:36 PM

Change 300889 had a related patch set uploaded (by ArielGlenn):
move generation of lists of good dumps to snapshot1007

https://gerrit.wikimedia.org/r/300889

ArielGlenn mentioned this in rOPUPfe4f261a3a49: move generation of lists of good dumps to snapshot1007.Jul 25 2016, 3:41 PM
gerritbot added a comment.Jul 25 2016, 3:42 PM

Change 300889 merged by ArielGlenn:
move generation of lists of good dumps to snapshot1007

https://gerrit.wikimedia.org/r/300889

ArielGlenn mentioned this in rOPUPe1312a2921b3: move generation of lists of good dumps to snapshot1007.Jul 25 2016, 3:47 PM
ArielGlenn updated the task description. (Show Details)Jul 25 2016, 3:47 PM
gerritbot added a comment.Jul 25 2016, 4:57 PM

Change 300903 had a related patch set uploaded (by ArielGlenn):
move wikidata json, ttl dumps cron job to snapshot1007

https://gerrit.wikimedia.org/r/300903

ArielGlenn mentioned this in rOPUP3e1a35187f85: move wikidata json, ttl dumps cron job to snapshot1007.Jul 25 2016, 5:01 PM
ArielGlenn mentioned this in rOPUP5a8303ea93ce: move wikidata json, ttl dumps cron job to snapshot1007.Jul 25 2016, 5:06 PM
gerritbot added a comment.Jul 25 2016, 5:08 PM

Change 300903 merged by ArielGlenn:
move wikidata json, ttl dumps cron job to snapshot1007

https://gerrit.wikimedia.org/r/300903

ArielGlenn updated the task description. (Show Details)Jul 25 2016, 5:27 PM
ArielGlenn updated the task description. (Show Details)Jul 26 2016, 10:29 AM
ArielGlenn updated the task description. (Show Details)
ArielGlenn updated the task description. (Show Details)Jul 27 2016, 8:46 AM
ArielGlenn moved this task from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.
ArielGlenn updated the task description. (Show Details)Jul 28 2016, 1:49 AM
ArielGlenn updated the task description. (Show Details)Jul 29 2016, 8:35 AM
ArielGlenn closed this task as Resolved.Jul 30 2016, 9:05 AM
ArielGlenn updated the task description. (Show Details)

All jobs run properly after move so this task can be closed.

ArielGlenn moved this task from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.Jul 30 2016, 9:05 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL