Page MenuHomePhabricator
Create Task
Maniphest T272836

Decom ms-be[1019-1026]
Closed, ResolvedPublic
Actions

Description

ms-be1019

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be1020

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.

ms-be1021

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be1022

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be1023

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be1024

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be1025

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

ms-be1026

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to DC ops team member depending on site of server.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
Decom ms-be[1019-1026]operations/puppetproduction+1 -57
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedfgiunchediT266016 Refresh and expand Swift hardware capacity
 Resolved• CmjohnsonT272836 Decom ms-be[1019-1026]
 DeclinedNoneT280961 Degraded RAID on ms-be1019

Event Timeline

fgiunchedi created this task.Jan 25 2021, 10:10 AM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jan 25 2021, 3:24 PM
Stashbot added a comment.Feb 8 2021, 1:54 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-08T13:54:53Z] <godog> swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836

Stashbot added a comment.Feb 9 2021, 9:22 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-09T09:22:34Z] <godog> swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836

fgiunchedi mentioned this in T272209: Degraded RAID on ms-be1032.Feb 9 2021, 4:03 PM
Stashbot added a comment.Feb 9 2021, 4:14 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-09T16:14:00Z] <godog> swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836

Stashbot added a comment.Feb 10 2021, 8:19 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-10T08:19:05Z] <godog> swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836

fgiunchedi mentioned this in T274488: ms-be1034 not powering on.Feb 15 2021, 10:15 AM
Stashbot added a comment.Feb 15 2021, 1:06 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-15T13:06:54Z] <godog> swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836

Stashbot added a comment.Feb 16 2021, 8:37 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-16T08:37:25Z] <godog> swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836

Stashbot added a comment.Mar 11 2021, 2:07 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-11T14:07:28Z] <godog> swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836

Stashbot mentioned this in T276193: Decom ms-be1034 from swift.Mar 15 2021, 8:33 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-15T08:33:42Z] <godog> swift eqiad-prod remove decom hosts from account/container rings - T272836 T276193

Stashbot added a comment.Mar 16 2021, 7:51 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-16T07:51:09Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] - T272836

MoritzMuehlenhoff subscribed.Mar 16 2021, 12:13 PM
Stashbot added a comment.Mar 17 2021, 7:50 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-17T07:50:14Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] - T272836

Stashbot added a comment.Mar 18 2021, 8:27 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-18T08:27:48Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] - T272836

Stashbot mentioned this in T268435: Add ms-be106[0-3] to swift.Mar 22 2021, 8:13 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-22T08:13:27Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836 T268435

Stashbot added a comment.Mar 29 2021, 7:42 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-29T07:42:05Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836 T268435

Stashbot added a comment.Apr 6 2021, 7:20 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-06T07:20:49Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836 T268435

Stashbot added a comment.Apr 14 2021, 7:41 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-14T07:41:10Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836

Stashbot added a comment.Apr 19 2021, 8:07 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-19T08:07:19Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836

fgiunchedi added a subtask: T280961: Degraded RAID on ms-be1019.Apr 23 2021, 8:19 AM
Stashbot added a comment.Apr 26 2021, 7:32 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-26T07:32:56Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836

Stashbot added a comment.Apr 27 2021, 7:26 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-27T07:26:57Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836

Stashbot added a comment.Apr 27 2021, 10:18 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-27T10:18:35Z] <godog> swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836

gerritbot added a comment.Apr 27 2021, 10:22 AM

Change 682920 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] Decom ms-be[1019-1026]

https://gerrit.wikimedia.org/r/682920

gerritbot added a project: Patch-For-Review.Apr 27 2021, 10:22 AM
gerritbot added a comment.Apr 27 2021, 12:42 PM

Change 682920 merged by Filippo Giunchedi:

[operations/puppet@production] Decom ms-be[1019-1026]

https://gerrit.wikimedia.org/r/682920

fgiunchedi renamed this task from Decom ms-be[1019-1026] from swift to Decom ms-be[1019-1026].Apr 27 2021, 12:47 PM
fgiunchedi updated the task description. (Show Details)
fgiunchedi updated the task description. (Show Details)
ops-monitoring-bot added a comment.Apr 27 2021, 1:05 PM

cookbooks.sre.hosts.decommission executed by filippo@cumin1001 for hosts: ms-be1019.eqiad.wmnet

  • ms-be1019.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Apr 27 2021, 1:10 PM
ops-monitoring-bot added a comment.Apr 27 2021, 1:21 PM

cookbooks.sre.hosts.decommission executed by filippo@cumin1001 for hosts: ms-be[1020-1026].eqiad.wmnet

  • ms-be1020.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1021.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1022.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1023.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1024.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1025.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1026.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
fgiunchedi added a project: ops-eqiad.Apr 27 2021, 1:23 PM
fgiunchedi added subscribers: Jclark-ctr, • Cmjohnson.

@Cmjohnson or @Jclark-ctr all yours, hosts ready for decom

fgiunchedi closed subtask T280961: Degraded RAID on ms-be1019 as Declined.Apr 27 2021, 1:23 PM
Maintenance_bot added a project: SRE.Apr 27 2021, 1:45 PM
fgiunchedi moved this task from Doing to Radar on the User-fgiunchedi board.Apr 27 2021, 1:46 PM
wiki_willy assigned this task to • Cmjohnson.Apr 29 2021, 12:02 AM
wiki_willy moved this task from Backlog to Decommission on the ops-eqiad board.
fgiunchedi mentioned this in Unknown Object (Task).Jun 29 2021, 8:30 PM
Jclark-ctr updated the task description. (Show Details)Jun 29 2021, 9:39 PM

Removed ms-be1026 to rack new host

fgiunchedi moved this task from Inbox to Radar on the SRE-swift-storage board.Aug 25 2021, 4:18 PM
Jclark-ctr updated the task description. (Show Details)Aug 25 2021, 7:48 PM
• Cmjohnson closed this task as Resolved.Sep 22 2021, 4:41 PM
• Cmjohnson updated the task description. (Show Details)

removed from rack and updated

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits