Page MenuHomePhabricator
Create Task
Maniphest T301251

Decommission ms-fe200[5-8]
Closed, ResolvedPublic
Actions

Description

ms-fe20[09-12] are now fully in service, so it's time to decommission ms-fe200[5-8].

Details

SubjectRepoBranchLines +/-
hieradata: move codfs swift::stats_reporter_hostoperations/puppetproduction+1 -1
swift: remove ms-fe200[5-8]operations/puppetproduction+0 -12
hieradata: move codfw swiftrepl host to ms-fe2009operations/puppetproduction+0 -0
Customize query in gerrit

Related Objects

Event Timeline

MatthewVernon created this task.Feb 8 2022, 3:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 8 2022, 3:15 PM
Stashbot added a comment.Feb 8 2022, 3:18 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-08T15:18:21Z] <Emperor> depooling ms-fe200[5-8] T301251

gerritbot added a comment.Feb 8 2022, 3:26 PM

Change 760961 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] hieradata: move codfw swiftrepl host to ms-fe2009

https://gerrit.wikimedia.org/r/760961

gerritbot added a project: Patch-For-Review.Feb 8 2022, 3:26 PM
gerritbot added a comment.Feb 8 2022, 4:08 PM

Change 760961 merged by MVernon:

[operations/puppet@production] hieradata: move codfw swiftrepl host to ms-fe2009

https://gerrit.wikimedia.org/r/760961

Maintenance_bot removed a project: Patch-For-Review.Feb 8 2022, 4:10 PM
MatthewVernon added a comment.Feb 8 2022, 4:32 PM

Nodes depooled, swiftrepl disabled (including removing the .timer file and swiftrepl.conf) on ms-fe2005.

MatthewVernon moved this task from Inbox to In progress on the SRE-swift-storage board.Feb 8 2022, 4:32 PM
gerritbot added a comment.Feb 9 2022, 9:19 AM

Change 761283 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: remove ms-fe200[5-8]

https://gerrit.wikimedia.org/r/761283

gerritbot added a project: Patch-For-Review.Feb 9 2022, 9:19 AM
gerritbot added a comment.Feb 9 2022, 9:45 AM

Change 761283 merged by MVernon:

[operations/puppet@production] swift: remove ms-fe200[5-8]

https://gerrit.wikimedia.org/r/761283

Stashbot added a comment.Feb 9 2022, 10:02 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-09T10:02:15Z] <Emperor> rolling restart of swift frontends T301251

Maintenance_bot removed a project: Patch-For-Review.Feb 9 2022, 10:10 AM
gerritbot added a comment.Feb 9 2022, 10:15 AM

Change 761293 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] hieradata: move codfs swift::stats_reporter_host

https://gerrit.wikimedia.org/r/761293

gerritbot added a project: Patch-For-Review.Feb 9 2022, 10:15 AM
gerritbot added a comment.Feb 9 2022, 10:30 AM

Change 761293 merged by MVernon:

[operations/puppet@production] hieradata: move codfs swift::stats_reporter_host

https://gerrit.wikimedia.org/r/761293

ops-monitoring-bot added a comment.Feb 9 2022, 11:08 AM

cookbooks.sre.hosts.decommission executed by mvernon@cumin2002 for hosts: ms-fe[2005-2008].codfw.wmnet

  • ms-fe2005.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-fe2006.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-fe2007.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-fe2008.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
MatthewVernon closed this task as Resolved.Feb 9 2022, 11:32 AM

This is now done - T301334 is the DC-team task to actually decommission the hardware.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL