Page MenuHomePhabricator
Create Task
Maniphest T272837

Decom ms-be[2016-2027]
Closed, ResolvedPublic
Actions

Description

ms-be2016.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2017.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2018.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2019.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2020.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2021.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2022.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2023.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2024.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2025.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2026.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be2027.codfw.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

Details

SubjectRepoBranchLines +/-
Decom ms-be[2016-2027]operations/puppetproduction+0 -76
autoinstall: Prepare ms-be2016 for reimage for swift backups testingoperations/puppetproduction+6 -1
hieradata: decom ms-be20[16-27] from swiftoperations/puppetproduction+0 -12
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedfgiunchediT266016 Refresh and expand Swift hardware capacity
 ResolvedPapaulT272837 Decom ms-be[2016-2027]

Event Timeline

fgiunchedi created this task.Jan 25 2021, 10:10 AM
Stashbot added a comment.Jan 25 2021, 10:44 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-25T10:44:13Z] <godog> swift decrease weight for ms-be20[16,18,20,22] - T272837

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jan 25 2021, 3:24 PM
Stashbot added a comment.Jan 26 2021, 8:31 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-26T08:30:59Z] <godog> swift start decom for ms-be20[17,19,21,23,24,25,26,27] - T272837

Stashbot added a comment.Jan 26 2021, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-26T14:03:25Z] <godog> swift codfw-prod decrease SSD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Jan 27 2021, 9:03 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-27T09:03:33Z] <godog> swift codfw-prod decrease SSD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Jan 27 2021, 1:20 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-27T13:19:59Z] <godog> swift codfw-prod decrease SSD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Jan 27 2021, 1:25 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-27T13:25:44Z] <godog> swift codfw-prod decrease SSD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Jan 28 2021, 8:34 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-28T08:34:20Z] <godog> swift codfw-prod decrease SSD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Jan 28 2021, 11:20 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-28T11:20:03Z] <godog> swift codfw-prod decrease SSD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Jan 28 2021, 1:17 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-28T13:17:24Z] <godog> swift codfw-prod decrease HDD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Feb 1 2021, 8:17 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-01T08:17:43Z] <godog> swift codfw-prod decrease HDD weight for ms-be20[16-27] - T272837

fgiunchedi mentioned this in T273453: ms-fe.svc.codfw.wmnet paged during Swift rebalance.Feb 1 2021, 1:27 PM
Stashbot added a comment.Feb 3 2021, 1:58 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-03T13:58:43Z] <godog> swift codfw-prod decrease HDD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Feb 4 2021, 2:38 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-04T14:38:42Z] <godog> swift codfw-prod decrease HDD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Feb 5 2021, 10:32 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-05T10:32:13Z] <godog> swift codfw-prod decrease HDD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Feb 8 2021, 8:33 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-08T08:33:27Z] <godog> swift codfw-prod decrease HDD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Feb 11 2021, 8:35 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-11T08:35:13Z] <godog> swift codfw-prod decrease HDD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Feb 16 2021, 5:24 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-16T17:24:19Z] <godog> swift codfw-prod decrease HDD weight for ms-be20[16-27] - T272837

Stashbot added a comment.Feb 18 2021, 3:06 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-18T15:06:51Z] <godog> swift codfw-prod decrease HDD weight for ms-be20[16-27] - T272837

gerritbot added a comment.Feb 19 2021, 8:11 AM

Change 665298 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: decom ms-be20[16-27] from swift

https://gerrit.wikimedia.org/r/665298

gerritbot added a project: Patch-For-Review.Feb 19 2021, 8:11 AM
gerritbot added a comment.Feb 19 2021, 2:50 PM

Change 665298 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: decom ms-be20[16-27] from swift

https://gerrit.wikimedia.org/r/665298

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2021, 3:10 PM
jcrespo added subscribers: LSobanski, jcrespo.EditedFeb 22 2021, 1:54 PM

I will be taking ms-be2016, after talking to @fgiunchedi, temporarily (probably just for a week) to test swift media production backups on codfw before ordering new ones at T264672. CC @LSobanski.

I intend to change the role and reimage it (including wiping its disks), and use it to store test backups. I was told they have been fully depooled from swift and firewalled its service.

ms-be2016 seems to be at the moment with healthy disks.

Please do not send it for decom. yet until after the performance test: T267338

jcrespo claimed this task.Feb 22 2021, 1:55 PM
gerritbot added a comment.Feb 24 2021, 1:45 PM

Change 666626 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] autoinstall: Prepare ms-be2016 for reimage for swift backups testing

https://gerrit.wikimedia.org/r/666626

gerritbot added a project: Patch-For-Review.Feb 24 2021, 1:45 PM
gerritbot added a comment.Feb 24 2021, 2:04 PM

Change 666626 merged by Jcrespo:
[operations/puppet@production] autoinstall: Prepare ms-be2016 for reimage for swift backups testing

https://gerrit.wikimedia.org/r/666626

ops-monitoring-bot added a comment.Feb 24 2021, 2:10 PM

Script wmf-auto-reimage was launched by jynus on cumin2001.codfw.wmnet for hosts:

ms-be2016.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102241410_jynus_16243_ms-be2016_codfw_wmnet.log.

Maintenance_bot removed a project: Patch-For-Review.Feb 24 2021, 2:10 PM
ops-monitoring-bot added a comment.Feb 24 2021, 2:36 PM

Completed auto-reimage of hosts:

['ms-be2016.codfw.wmnet']

and were ALL successful.

fgiunchedi mentioned this in T275752: Jobrunner timeouts on cross-DC file uploads because of HTTP/2.Feb 26 2021, 10:14 AM
jcrespo reassigned this task from jcrespo to fgiunchedi.Mar 5 2021, 3:35 PM

I will no longer be using ms-be2016 or any other host. They can be freely decommissioned. Let me know if you need help with any of the tasks involving them.

gerritbot added a comment.Mar 9 2021, 7:37 AM

Change 670086 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Decom ms-be[2016-2027]

https://gerrit.wikimedia.org/r/670086

gerritbot added a project: Patch-For-Review.Mar 9 2021, 7:37 AM
gerritbot added a comment.Mar 9 2021, 7:39 AM

Change 670086 merged by Filippo Giunchedi:
[operations/puppet@production] Decom ms-be[2016-2027]

https://gerrit.wikimedia.org/r/670086

fgiunchedi updated the task description. (Show Details)Mar 9 2021, 7:58 AM
ops-monitoring-bot added a comment.Mar 9 2021, 8:26 AM

cookbooks.sre.hosts.decommission executed by filippo@cumin1001 for hosts: ms-be2016.codfw.wmnet

  • ms-be2016.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Mar 9 2021, 8:59 AM

cookbooks.sre.hosts.decommission executed by filippo@cumin1001 for hosts: ms-be[2017-2019].codfw.wmnet

  • ms-be2017.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be2018.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be2019.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Mar 9 2021, 10:43 AM

cookbooks.sre.hosts.decommission executed by filippo@cumin1001 for hosts: ms-be[2020-2027].codfw.wmnet

  • ms-be2020.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be2021.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be2022.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be2023.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be2024.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be2025.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be2026.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be2027.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
fgiunchedi reassigned this task from fgiunchedi to Papaul.Mar 9 2021, 10:44 AM
fgiunchedi edited projects, added ops-codfw; removed Patch-For-Review, SRE-swift-storage.
fgiunchedi added a subscriber: Papaul.

@Papaul all yours

fgiunchedi renamed this task from Decom ms-be[2016-2027] from swift to Decom ms-be[2016-2027].Mar 9 2021, 10:44 AM
Maintenance_bot added a project: SRE.Mar 9 2021, 10:45 AM
Papaul triaged this task as Medium priority.Mar 9 2021, 3:58 PM

@fgiunchedi thank you

Papaul updated the task description. (Show Details)Mar 9 2021, 6:35 PM
Papaul updated the task description. (Show Details)Mar 9 2021, 10:55 PM
Papaul closed this task as Resolved.Mar 10 2021, 5:04 PM
Papaul updated the task description. (Show Details)

complete

jbond mentioned this in T185239: Puppet hosts with signed certificate present on agent but not master.Aug 4 2022, 11:34 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL