Page MenuHomePhabricator
Create Task
Maniphest T182540

get datset1001, ms1001 ready for decommission
Closed, ResolvedPublic
Actions

Description

These boxes are well out of warranty so they need to go away. Steps required for this to happen:

  • snapshot jobs moved off these hosts
  • mounts removed from snapshot hosts
  • web service moved to labstore1006,7 (T188641, T188646)
  • nfs mounts to stat1005,6 moved to labstore1006,7 (T188644)
  • incoming fetches of datasets moved to labstore1006,7
  • incoming pushes of datasets from other hosts moved to labstore1006,7 (T171541)
  • ezachte's cron jobs updated to use labstore1006,7 instead of dataset1001
  • get all mirror sites to switch over to labstore1006,7 (T188645)
  • service of dumps and pageviews via labstore1006,7 instead of labstore1003 (T188643)
  • turn off all cron jobs involving dataset1001, ms1001
  • remove all mention of dataset1001, ms1001 from rsync config files
  • change roles on dataset1001, ms1001 to spare system
  • Profit??

This may not be quite the complete list but it should be most items. Some of these items are also listed on this task: T168486
I have them here for my own bookkeeping.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedArielGlennT182540 get datset1001, ms1001 ready for decommission
 Resolved• CmjohnsonT186756 Move labstore1006 and 1007 to 10G enabled racks in row A & D
 Resolved• madhuvishyT168486 Migrate customer-facing Dumps endpoints to Cloud Services
 Resolved• madhuvishyT171508 Investigate and implement alternative for showmount based check at instance boot time
 Resolved• madhuvishyT174590 Revert patch that adds a temporary exception to the block-for-export check for the testlabs project
 Resolved• madhuvishyT171539 Puppetize and setup initial lvms and directory structures for labstore1006|7
 ResolvedArielGlennT171541 Setup periodic rsync jobs from dumps generation hosts to labstore1006|7
 ResolvedArielGlennT188726 make sure all datasets in xmldatadumps/public/other on dataset1001 are accounted for on new labs boxes
 Resolved• ezachteT189283 Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches
 ResolvedArielGlennT189284 Stop serving slowparse logs from dumps distribution servers
 Resolved• madhuvishyT181431 Setup NFS on dumps servers
 Resolved• madhuvishyT171540 Figure out how NFS failovers will work for the dumps servers - labstore1006|7
 Resolved• madhuvishyT188073 Maintain symlinks for WMCS NFS Dumps users with new directory structure
 Resolved• madhuvishyT188641 Set up the web service that serves dumps.wikimedia.org
 Resolved• madhuvishyT188642 Set up labstore1006|7 as the source for rsync mirror sites
 Resolved• madhuvishyT188643 Migrate Dumps WMCS NFS users from labstore1003 to labstore1006/7
 Resolved• madhuvishyT188644 Migrate the stat* mount from dataset1001 to labstore1006/7
 Resolved• madhuvishyT188645 Get all the rsync mirror sites to to switch over to labstore1006,7
 Resolved• madhuvishyT188646 Point dumps.wikimedia.org to labstore1006/7
 Resolved• madhuvishyT185101 Labstore1006/7 profile for meltdown kernel
 Resolved• madhuvishyT188647 Announce/Communicate dumps migration to labstore1006|7 to stakeholders
Restricted Task

Event Timeline

ArielGlenn triaged this task as Medium priority.Dec 10 2017, 8:00 PM
ArielGlenn created this task.
ArielGlenn updated the task description. (Show Details)
gerritbot subscribed.Dec 10 2017, 8:02 PM

Change 396931 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] get rid of datasets1001 mount on snapshot hosts

https://gerrit.wikimedia.org/r/396931

gerritbot added a project: Patch-For-Review.Dec 10 2017, 8:02 PM
ArielGlenn moved this task from Backlog to Active on the Dumps-Generation board.Dec 10 2017, 8:02 PM
gerritbot added a comment.Dec 12 2017, 9:47 AM

Change 396931 merged by ArielGlenn:
[operations/puppet@production] get rid of datasets1001 mount on snapshot hosts

https://gerrit.wikimedia.org/r/396931

ArielGlenn updated the task description. (Show Details)Dec 12 2017, 12:20 PM
ArielGlenn moved this task from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.Jan 16 2018, 12:17 PM
ArielGlenn added subtasks: T186756: Move labstore1006 and 1007 to 10G enabled racks in row A & D, T171541: Setup periodic rsync jobs from dumps generation hosts to labstore1006|7, T181431: Setup NFS on dumps servers, T168486: Migrate customer-facing Dumps endpoints to Cloud Services.Feb 8 2018, 9:53 AM
• madhuvishy closed subtask T186756: Move labstore1006 and 1007 to 10G enabled racks in row A & D as Resolved.Feb 20 2018, 4:27 PM
• madhuvishy updated the task description. (Show Details)Mar 1 2018, 6:11 PM
• madhuvishy closed subtask T171541: Setup periodic rsync jobs from dumps generation hosts to labstore1006|7 as Resolved.Mar 5 2018, 5:58 PM
• madhuvishy closed subtask T181431: Setup NFS on dumps servers as Resolved.Mar 16 2018, 12:19 AM
gerritbot added a comment.Apr 3 2018, 4:25 PM

Change 423731 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] dumps: Turn off cron that rsyncs to labstore1003

https://gerrit.wikimedia.org/r/423731

gerritbot added a comment.Apr 3 2018, 4:32 PM

Change 423732 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] dumps: Clean up code that rsyncs to labstore1003

https://gerrit.wikimedia.org/r/423732

gerritbot added a comment.Apr 3 2018, 4:39 PM

Change 423733 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] dumps: Remove stat1005|6 from nfs clients for dataset1001

https://gerrit.wikimedia.org/r/423733

gerritbot added a comment.Apr 10 2018, 9:50 AM

Change 425234 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] remove dumps web server from dataset1001 and ms1001

https://gerrit.wikimedia.org/r/425234

gerritbot added a comment.Apr 10 2018, 11:03 AM

Change 425246 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] turn off public dumps mirror rsync access to dataset1001

https://gerrit.wikimedia.org/r/425246

gerritbot added a comment.Apr 10 2018, 5:32 PM

Change 425234 merged by ArielGlenn:
[operations/puppet@production] remove dumps web server from dataset1001 and ms1001

https://gerrit.wikimedia.org/r/425234

gerritbot added a comment.Apr 10 2018, 5:36 PM

Change 425246 merged by ArielGlenn:
[operations/puppet@production] turn off public dumps mirror rsync access to dataset1001

https://gerrit.wikimedia.org/r/425246

ArielGlenn moved this task from Blocked/Stalled/Waiting for event to Active on the Dumps-Generation board.Apr 10 2018, 5:51 PM
gerritbot added a comment.Apr 11 2018, 5:27 PM

Change 423733 merged by Madhuvishy:
[operations/puppet@production] dumps: Remove stat1005|6 from nfs clients for dataset1001

https://gerrit.wikimedia.org/r/423733

gerritbot added a comment.Apr 11 2018, 6:17 PM

Change 423731 merged by Madhuvishy:
[operations/puppet@production] dumps: Turn off cron that rsyncs to labstore1003

https://gerrit.wikimedia.org/r/423731

gerritbot added a comment.Apr 11 2018, 6:25 PM

Change 423732 merged by Madhuvishy:
[operations/puppet@production] dumps: Clean up code that rsyncs to labstore1003

https://gerrit.wikimedia.org/r/423732

ArielGlenn updated the task description. (Show Details)Apr 11 2018, 7:06 PM
gerritbot added a comment.Apr 12 2018, 8:55 AM

Change 425773 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] remove ms1001 from dump rsync targets

https://gerrit.wikimedia.org/r/425773

gerritbot added a comment.Apr 12 2018, 9:05 AM

Change 425773 merged by ArielGlenn:
[operations/puppet@production] remove ms1001 from dump rsync targets

https://gerrit.wikimedia.org/r/425773

gerritbot added a comment.Apr 12 2018, 9:20 AM

Change 425776 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] Turn off incoming rsyncs of various datasets to dataset1001

https://gerrit.wikimedia.org/r/425776

gerritbot added a comment.Apr 12 2018, 9:22 AM

Change 425776 merged by ArielGlenn:
[operations/puppet@production] Turn off incoming rsyncs of various datasets to dataset1001

https://gerrit.wikimedia.org/r/425776

ArielGlenn updated the task description. (Show Details)Apr 13 2018, 11:07 AM
• madhuvishy closed subtask T168486: Migrate customer-facing Dumps endpoints to Cloud Services as Resolved.Apr 24 2018, 11:57 PM
ArielGlenn updated the task description. (Show Details)Apr 30 2018, 7:00 AM
gerritbot added a comment.Apr 30 2018, 7:09 AM

Change 429724 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] remove all references to dataset1001 from rsync and related manifests

https://gerrit.wikimedia.org/r/429724

gerritbot added a comment.Apr 30 2018, 7:27 AM

Change 429724 merged by ArielGlenn:
[operations/puppet@production] remove all references to dataset1001 from rsync and related manifests

https://gerrit.wikimedia.org/r/429724

gerritbot added a comment.Apr 30 2018, 7:32 AM

Change 429726 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] stop rsync of web logs from dataset1001

https://gerrit.wikimedia.org/r/429726

gerritbot added a comment.Apr 30 2018, 7:35 AM

Change 429726 merged by ArielGlenn:
[operations/puppet@production] stop rsync of web logs from dataset1001

https://gerrit.wikimedia.org/r/429726

gerritbot added a comment.Apr 30 2018, 7:43 AM

Change 429728 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] remove production roles from ms1001, dataset1001

https://gerrit.wikimedia.org/r/429728

ArielGlenn updated the task description. (Show Details)Apr 30 2018, 7:44 AM
gerritbot added a comment.May 7 2018, 10:16 AM

Change 429728 merged by ArielGlenn:
[operations/puppet@production] remove production roles from ms1001, dataset1001

https://gerrit.wikimedia.org/r/429728

ArielGlenn updated the task description. (Show Details)May 7 2018, 10:51 AM
gerritbot added a comment.May 7 2018, 10:55 AM

Change 431551 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] remove a bunch of manifests only used on dataset1001, ms1001

https://gerrit.wikimedia.org/r/431551

gerritbot added a comment.May 7 2018, 11:01 AM

Change 431551 merged by ArielGlenn:
[operations/puppet@production] remove a bunch of manifests only used on dataset1001, ms1001

https://gerrit.wikimedia.org/r/431551

ArielGlenn closed this task as Resolved.May 7 2018, 4:53 PM
ArielGlenn updated the task description. (Show Details)
ArielGlenn moved this task from Active to Done on the Dumps-Generation board.May 8 2018, 7:49 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits