Page MenuHomePhabricator
Create Task
Maniphest T188644

Migrate the stat* mount from dataset1001 to labstore1006/7
Closed, ResolvedPublic
Actions

Description

Task for actual migration, need to coordinate with Analytics/Ezachte

Details

SubjectRepoBranchLines +/-
dumps: Remove stat1005|6 from nfs clients for dataset1001operations/puppetproduction+0 -3
statistics: Symlink /mnt/data to nfs share from active serveroperations/puppetproduction+6 -13
statistics: Absent existing dumps mount at /mnt/dataoperations/puppetproduction+2 -2
statistics: Add missed README for dumps nfs mountoperations/puppetproduction+5 -0
statistics: Mount dumps share from labstore1006|7 on stat1005|6operations/puppetproduction+51 -7
Customize query in gerrit

Related Objects
Search...

Event Timeline

• madhuvishy triaged this task as Medium priority.Mar 1 2018, 6:07 PM
• madhuvishy created this task.
• madhuvishy mentioned this in T182540: get datset1001, ms1001 ready for decommission.Mar 1 2018, 6:11 PM
• madhuvishy mentioned this in T168486: Migrate customer-facing Dumps endpoints to Cloud Services.
• madhuvishy mentioned this in T188647: Announce/Communicate dumps migration to labstore1006|7 to stakeholders.Mar 7 2018, 10:03 PM
bd808 edited projects, added cloud-services-team (Kanban); removed cloud-services-team (FY2017-18), Goal.Mar 9 2018, 4:31 PM
• madhuvishy added a subtask: T176091: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet).Mar 16 2018, 12:52 AM
• madhuvishy mentioned this in T176091: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet).
• madhuvishy mentioned this in T171540: Figure out how NFS failovers will work for the dumps servers - labstore1006|7.Mar 16 2018, 5:44 PM
gerritbot subscribed.Mar 16 2018, 5:55 PM

Change 420083 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] [WIP] statistics: Migrate dumps mount to labstore1006|7

https://gerrit.wikimedia.org/r/420083

gerritbot added a project: Patch-For-Review.Mar 16 2018, 5:55 PM
gerritbot added a comment.Mar 29 2018, 8:07 AM

Change 422892 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] statistics: Absent existing dumps mount at /mnt/data

https://gerrit.wikimedia.org/r/422892

gerritbot added a comment.Mar 29 2018, 8:12 AM

Change 422896 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] statistics: Symlink /mnt/data to nfs share from active server

https://gerrit.wikimedia.org/r/422896

gerritbot added a comment.Apr 2 2018, 6:12 PM

Change 420083 merged by Madhuvishy:
[operations/puppet@production] statistics: Mount dumps share from labstore1006|7 on stat1005|6

https://gerrit.wikimedia.org/r/420083

gerritbot added a comment.Apr 2 2018, 6:16 PM

Change 423515 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] statistics: Add missed README for dumps nfs mount

https://gerrit.wikimedia.org/r/423515

gerritbot added a comment.Apr 2 2018, 6:16 PM

Change 423515 merged by Madhuvishy:
[operations/puppet@production] statistics: Add missed README for dumps nfs mount

https://gerrit.wikimedia.org/r/423515

gerritbot added a comment.Apr 2 2018, 6:21 PM

Change 422892 merged by Madhuvishy:
[operations/puppet@production] statistics: Absent existing dumps mount at /mnt/data

https://gerrit.wikimedia.org/r/422892

gerritbot added a comment.Apr 2 2018, 6:25 PM

Change 422896 merged by Madhuvishy:
[operations/puppet@production] statistics: Symlink /mnt/data to nfs share from active server

https://gerrit.wikimedia.org/r/422896

• madhuvishy added a comment.Apr 2 2018, 6:34 PM

Notes from migration plan:

  • Announcements
    • [DONE] Last minute analytics mailing list update
  • Silence monitoring -- need to check if there's anything set up on stat*
    • [DONE] Downtimed puppet run alerts for stat1005|6
  • [DONE] Mount NFS shares from labstore1006 & 7 on stat1005|6 at /mnt/nfs/labstore1006-dumps & /mnt/nfs/labstore1007-dumps
  • [NONE RUNNING] Kill any processes actively accessing /mnt/data
    • find with lsof +f -- /mnt/data
  • [DONE] Absent NFS mount at /mnt/data (served from dataset1001)
    • Disable puppet on stat1005|6
    • Merge puppet patch - https://gerrit.wikimedia.org/r/#/c/422892/
    • Apply patch on stat* and test
      • Puppet refused to remove directory /mnt/data without --force. Manually ran -- madhuvishy@stat1005:/mnt$ sudo rm -r data

Success Criteria:

  • [SUCCESS] stat1005 & 6 can sucessfully read from /mnt/data
    • head -n 1 /mnt/data/xmldatadumps/public/liwiki/latest/liwiki-latest-pages-articles.xml.bz2-rss.xml
  • Labstore1007 - load on the server is normal (monitor for a couple hours atleast) -- All good right now, nothing much is going on though

Post (if success):

Rollback

  • Kill any processes actively accessing /mnt/data
    • find with lsof +f -- /mnt/data
  • [POSSIBLY UNDO] Mount NFS shares from labstore1006 & 7 on stat1005|6 at /mnt/nfs/labstore1006-dumps & /mnt/nfs/labstore1007-dumps
• madhuvishy added a comment.Apr 2 2018, 7:33 PM

This went well. Clean up task pending: Remove the dumps NFS export from dataset1001

gerritbot added a comment.Apr 3 2018, 4:39 PM

Change 423733 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] dumps: Remove stat1005|6 from nfs clients for dataset1001

https://gerrit.wikimedia.org/r/423733

• madhuvishy added a comment.Apr 3 2018, 4:47 PM
In T188644#4098287, @madhuvishy wrote:

This went well. Clean up task pending: Remove the dumps NFS export from dataset1001

Patch up - https://gerrit.wikimedia.org/r/423733

• madhuvishy removed a subtask: T176091: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet).Apr 4 2018, 6:13 PM
gerritbot added a comment.Apr 11 2018, 5:27 PM

Change 423733 merged by Madhuvishy:
[operations/puppet@production] dumps: Remove stat1005|6 from nfs clients for dataset1001

https://gerrit.wikimedia.org/r/423733

• madhuvishy closed this task as Resolved.Apr 11 2018, 6:07 PM
bd808 moved this task from Inbox to Done on the cloud-services-team (Kanban) board.May 6 2018, 6:48 PM
ArielGlenn moved this task from Backlog to Done on the Datasets-General-or-Unknown board.Jul 2 2018, 12:37 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits