Page MenuHomePhabricator
Create Task
Maniphest T337269

decommission labstore100[45].eqiad.wmne
Closed, ResolvedPublicRequest
Actions

Description

This task will track the decommission-hardware of servers labstore1004 and labstore1005.

With the launch of updates to the decom cookbook, the majority of these steps can be handled by the service owners directly. The DC Ops team only gets involved once the system has been fully removed from service and powered down by the decommission cookbook.

labstore1004.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member and site project (ops-sitename) depending on site of server

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.

labstore1005.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member and site project (ops-sitename) depending on site of server

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
Remove references to labstore100[45]operations/puppetproduction+2 -11
Customize query in gerrit

Related Objects

Event Timeline

Andrew created this task.May 22 2023, 8:29 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptMay 22 2023, 8:29 PM
gerritbot added a comment.May 22 2023, 8:36 PM

Change 922168 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Remove references to labstore100[45]

https://gerrit.wikimedia.org/r/922168

gerritbot added a project: Patch-For-Review.May 22 2023, 8:36 PM
Andrew updated the task description. (Show Details)May 22 2023, 8:37 PM
gerritbot added a comment.May 22 2023, 8:39 PM

Change 922168 merged by Andrew Bogott:

[operations/puppet@production] Remove references to labstore100[45]

https://gerrit.wikimedia.org/r/922168

ops-monitoring-bot added a comment.May 22 2023, 8:44 PM

cookbooks.sre.hosts.decommission executed by andrew@cumin1001 for hosts: labstore1005.eqiad.wmnet

  • labstore1005.eqiad.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Host steps raised exception: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

Andrew assigned this task to Jclark-ctr.May 22 2023, 8:48 PM
Andrew updated the task description. (Show Details)
Andrew added a project: ops-eqiad.

Two things:

  1. The decom script said this during both runs:
Traceback (most recent call last):
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/decommission.py", line 429, in run
    self._decommission_host(fqdn)
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/decommission.py", line 370, in _decommission_host
    configure_switch_interfaces(self.remote, netbox, netbox_data, self.spicerack.verbose)
  File "/srv/deployment/spicerack/cookbooks/sre/network/__init__.py", line 40, in configure_switch_interfaces
    switch_fqdn = nb_switch_interface.device.primary_ip.dns_name
AttributeError: 'NoneType' object has no attribute 'dns_name'
  1. These hosts likely have drive shelves attached to them that should also be removed.
ops-monitoring-bot added a comment.May 22 2023, 8:54 PM

cookbooks.sre.hosts.decommission executed by andrew@cumin1001 for hosts: labstore1004.eqiad.wmnet

  • labstore1004.eqiad.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Host steps raised exception: 'NoneType' object has no attribute 'dns_name'

ERROR: some step on some host failed, check the bolded items above

Maintenance_bot removed a project: Patch-For-Review.May 22 2023, 9:11 PM
Maintenance_bot added a project: SRE.May 22 2023, 9:29 PM
Jclark-ctr closed this task as Resolved.May 24 2023, 1:28 PM
Jclark-ctr updated the task description. (Show Details)
Stashbot added a comment.May 31 2023, 8:52 AM

Mentioned in SAL (#wikimedia-operations) [2023-05-31T08:52:40Z] <moritzm> manually run puppet node clean/deactivate for labstore1004/1005 (which run into a traceback in the decom script) T337269

ops-monitoring-bot added a comment.May 31 2023, 8:55 AM

Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: labstore1004.eqiad.wmnet

ops-monitoring-bot added a comment.May 31 2023, 8:55 AM

Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: labstore1005.eqiad.wmnet

MoritzMuehlenhoff mentioned this in T337003: Decommission labstore1004 and labstore1005 once they're no longer used.May 31 2023, 8:56 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL