Page MenuHomePhabricator
Create Task
Maniphest T335777

Q4:rack/decom codfw unified decommission task
Open, In Progress, LowPublic
Actions

Description

lvs2007

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - mgmt dns entries removed.

lvs2008

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - mgmt dns entries removed.

lvs2009

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - mgmt dns entries removed.

lvs2010

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - mgmt dns entries removed.

dns

dns2001

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - mgmt dns entries removed.

dns2002

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - mgmt dns entries removed.

dns2003

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - mgmt dns entries removed.

Details

Show related patches Customize query in gerrit

Event Timeline

ssingh created this task.Tue, May 2, 1:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, May 2, 1:54 PM
ssingh triaged this task as Low priority.Tue, May 2, 1:55 PM
gerritbot added a comment.Tue, May 2, 2:10 PM

Change 914341 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] lvs2007: decommission host for codfw hardware refresh

https://gerrit.wikimedia.org/r/914341

gerritbot added a project: Patch-For-Review.Tue, May 2, 2:10 PM
gerritbot added a comment.Tue, May 2, 2:15 PM

Change 914343 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] depool codfw (emergency patch, do not merge)

https://gerrit.wikimedia.org/r/914343

gerritbot added a comment.Tue, May 2, 2:18 PM

Change 914344 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: remove decommissioned host lvs2007 from lvs_neighbors

https://gerrit.wikimedia.org/r/914344

Stashbot added a comment.Wed, May 3, 2:33 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-03T14:33:17Z] <sukhe> set routing-options static route 208.80.153.224/28 next-hop 10.192.49.7 [move static route for high-traffic1 to lvs2010]: T335777

ops-monitoring-bot added a comment.Wed, May 3, 2:46 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: lvs2007.codfw.wmnet

  • lvs2007.codfw.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Wed, May 3, 2:49 PM

Change 914341 merged by Ssingh:

[operations/puppet@production] lvs2007: decommission host for codfw hardware refresh

https://gerrit.wikimedia.org/r/914341

gerritbot added a comment.Wed, May 3, 2:50 PM

Change 914344 merged by Ssingh:

[operations/homer/public@master] sites.yaml: remove decommissioned host lvs2007 from lvs_neighbors

https://gerrit.wikimedia.org/r/914344

ssingh updated the task description. (Show Details)Wed, May 3, 2:51 PM
Stashbot added a comment.Wed, May 3, 2:52 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-03T14:52:55Z] <sukhe> homer "cr*-codfw*" commit "Gerrit: 914344 remove decommissioned host lvs2007": T335777

Stashbot added a comment.Wed, May 3, 2:54 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-03T14:54:12Z] <sukhe> [finished] homer "cr*-codfw*" commit "Gerrit: 914344 remove decommissioned host lvs2007": T335777

Papaul added subscribers: Jhancock.wm, Papaul.Wed, May 3, 5:55 PM

@Jhancock.wm can you run the netbox offline script and get lvs2007 out of the rack and into storage ?
Thanks

Jhancock.wm updated the task description. (Show Details)Fri, May 5, 2:37 PM
Maintenance_bot added a project: SRE.Fri, May 5, 2:45 PM
ssingh updated the task description. (Show Details)Mon, May 8, 1:50 PM
ssingh updated the task description. (Show Details)
Stashbot added a comment.Mon, May 8, 3:32 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-08T15:32:10Z] <sukhe> ns1: remove dns2001, add dns2004 next-hop [ 208.80.153.48 208.80.153.111 208.80.153.10 ]: T335777

gerritbot added a comment.Mon, May 8, 3:47 PM

Change 917364 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: remove dns2001 from anycast_neighbors (host decom)

https://gerrit.wikimedia.org/r/917364

gerritbot added a comment.Mon, May 8, 3:52 PM

Change 917365 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: decommission dns2001

https://gerrit.wikimedia.org/r/917365

Papaul assigned this task to ssingh.Tue, May 9, 1:19 PM
Papaul moved this task from Backlog to Decommission on the ops-codfw board.
gerritbot added a comment.Tue, May 9, 1:30 PM

Change 917882 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] lvs2008: decommission host for codfw hardware refresh

https://gerrit.wikimedia.org/r/917882

gerritbot added a comment.Tue, May 9, 1:34 PM

Change 917885 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: remove decommissioned host lvs2008 from lvs_neighbors

https://gerrit.wikimedia.org/r/917885

Stashbot added a comment.Tue, May 9, 2:15 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-09T14:15:21Z] <sukhe> set routing-options static route 208.80.153.240/28 next-hop 10.192.49.7 [move static route for high-traffic2 to lvs2010]: T335777

ops-monitoring-bot added a comment.Tue, May 9, 2:45 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: lvs2008.codfw.wmnet

  • lvs2008.codfw.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Tue, May 9, 2:46 PM

Change 917882 merged by Ssingh:

[operations/puppet@production] lvs2008: decommission host for codfw hardware refresh

https://gerrit.wikimedia.org/r/917882

gerritbot added a comment.Tue, May 9, 2:47 PM

Change 917885 merged by Ssingh:

[operations/homer/public@master] sites.yaml: remove decommissioned host lvs2008 from lvs_neighbors

https://gerrit.wikimedia.org/r/917885

ssingh updated the task description. (Show Details)Tue, May 9, 2:51 PM
BCornwall changed the task status from Open to In Progress.Wed, May 10, 7:20 PM
BCornwall moved this task from Backlog to Traffic team actively servicing on the Traffic board.
gerritbot added a comment.Fri, May 12, 1:56 PM

Change 917365 abandoned by Ssingh:

[operations/puppet@production] hiera: decommission dns2001

Reason:

probably best to start clean

https://gerrit.wikimedia.org/r/917365

gerritbot added a comment.Fri, May 12, 2:07 PM

Change 919340 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: decommission dns2001

https://gerrit.wikimedia.org/r/919340

gerritbot added a comment.Fri, May 12, 2:13 PM

Change 917364 merged by Ssingh:

[operations/homer/public@master] sites.yaml: remove dns2001 from anycast_neighbors (host decom)

https://gerrit.wikimedia.org/r/917364

Stashbot added a comment.Fri, May 12, 2:13 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-12T14:13:40Z] <sukhe> homer "cr*-codfw*" commit "Gerrit: 917364 remove to-be decommissioned host dns2001": T335777

Stashbot added a comment.Fri, May 12, 2:15 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-12T14:15:55Z] <sukhe> [done] homer "cr*-codfw*" commit "Gerrit: 917364 remove to-be decommissioned host dns2001": T335777

gerritbot added a comment.Fri, May 12, 2:16 PM

Change 919340 merged by Ssingh:

[operations/puppet@production] hiera: decommission dns2001

https://gerrit.wikimedia.org/r/919340

ssingh updated the task description. (Show Details)Fri, May 12, 2:28 PM
ops-monitoring-bot added a comment.Fri, May 12, 2:35 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: dns2001.wikimedia.wmnet

  • dns2001.wikimedia.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Fri, May 12, 3:02 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: dns2001.wikimedia.org

  • dns2001.wikimedia.org (FAIL)
    • Unable to find/resolve the mgmt DNS record, using the IP instead: 10.193.1.197
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Host is already powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Tue, May 16, 4:05 PM

Change 920320 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: remove dns2002 from anycast_neighbors (host decom)

https://gerrit.wikimedia.org/r/920320

gerritbot added a comment.Tue, May 16, 4:56 PM

Change 920350 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: decommission dns2002

https://gerrit.wikimedia.org/r/920350

gerritbot added a comment.Tue, May 16, 4:58 PM

Change 920320 merged by Ssingh:

[operations/homer/public@master] sites.yaml: remove dns2002 from anycast_neighbors (host decom)

https://gerrit.wikimedia.org/r/920320

Stashbot added a comment.Tue, May 16, 5:00 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-16T17:00:13Z] <sukhe> homer "cr*-codfw*" commit "Gerrit: 920320 remove to-be decommissioned host dns2002" T335777

gerritbot added a comment.Tue, May 16, 5:05 PM

Change 920350 merged by Ssingh:

[operations/puppet@production] hiera: decommission dns2002

https://gerrit.wikimedia.org/r/920350

ops-monitoring-bot added a comment.Tue, May 16, 5:18 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: dns2002.wikimedia.org

  • dns2002.wikimedia.org (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ssingh updated the task description. (Show Details)Tue, May 16, 5:19 PM
gerritbot added a comment.Tue, May 16, 5:20 PM

Change 920355 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: remove obsolete dns2001.yaml file

https://gerrit.wikimedia.org/r/920355

gerritbot added a comment.Tue, May 16, 5:21 PM

Change 920355 merged by Ssingh:

[operations/puppet@production] hiera: remove obsolete dns2001.yaml file

https://gerrit.wikimedia.org/r/920355

gerritbot added a comment.Tue, May 16, 6:39 PM

Change 920363 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: remove dns2003 from anycast_neighbors (host decom)

https://gerrit.wikimedia.org/r/920363

gerritbot added a comment.Tue, May 16, 6:42 PM

Change 920364 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: decommission dns2003

https://gerrit.wikimedia.org/r/920364

gerritbot added a comment.Tue, May 16, 6:44 PM

Change 920363 merged by jenkins-bot:

[operations/homer/public@master] sites.yaml: remove dns2003 from anycast_neighbors (host decom)

https://gerrit.wikimedia.org/r/920363

Stashbot added a comment.Tue, May 16, 6:46 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-16T18:46:42Z] <sukhe> homer "cr*-codfw*" commit "Gerrit: 920363 remove to-be decommissioned host dns2003": T335777

gerritbot added a comment.Tue, May 16, 6:48 PM

Change 920364 merged by Ssingh:

[operations/puppet@production] hiera: decommission dns2003

https://gerrit.wikimedia.org/r/920364

ops-monitoring-bot added a comment.Tue, May 16, 7:00 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: dns2003.wikimedia.org

  • dns2003.wikimedia.org (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ssingh updated the task description. (Show Details)Tue, May 16, 7:01 PM
Jhancock.wm added a comment.Wed, May 17, 2:25 PM

@ssingh is it safe for me to physically remove lvs2008 and the three dns servers from the racks and offline them?

ssingh added a comment.Wed, May 17, 2:49 PM
In T335777#8859472, @Jhancock.wm wrote:

@ssingh is it safe for me to physically remove lvs2008 and the three dns servers from the racks and offline them?

Yes thanks! It's safe to remove lvs2008 and also dns200[1-3]. All hosts have been decommissioned from our end and are not in production.

Jhancock.wm updated the task description. (Show Details)Wed, May 17, 3:48 PM
gerritbot added a comment.Wed, May 17, 5:42 PM

Change 914343 abandoned by Ssingh:

[operations/dns@master] depool codfw (emergency patch, do not merge)

Reason:

no longer required

https://gerrit.wikimedia.org/r/914343

Maintenance_bot removed a project: Patch-For-Review.Wed, May 17, 6:12 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL