This task will track the #decommission of servers `cloudelastic100[1-4].wikimedia.org`
Note:
Decom cookbooks were ran, but there was one small failure:`cloudelastic1001` probably failed to fully wipe its disks, see https://phabricator.wikimedia.org/P57418:
```
wipefs: error: /dev/sdg: probing initialization failed: Read-only file system
wipefs: error: /dev/sdh: probing initialization failed: Read-only file system
wipefs: error: /dev/sdi: probing initialization failed: Read-only file system
/dev/sdj: 2 bytes were erased at offset 0x000001fe (dos): 55 aa
wipefs: error: /dev/sdj1: probing initialization failed: No such device or address
wipefs: error: /dev/sdk: probing initialization failed: Read-only file system
/dev/sdl: 2 bytes were erased at offset 0x000001fe (dos): 55 aa
wipefs: error: /dev/sdl1: probing initialization failed: No such device or address
**Failed to wipe swraid, partition-table and filesystem signatures, manual intervention required to make it unbootable**: Cumin execution failed (exit_code=2)
```
`cloudelastic1001.wikimedia.org`
**Steps for service owner:**
[x] - all system services confirmed offline from production use
[x] - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
[x] - remove system from all lvs/pybal active configuration
[x] - any service group puppet/hiera/dsh config removed
[x] - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
[x] - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
[x] - remove all remaining puppet references and all host entries in the puppet repo
[x] - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.
**End service owner steps / Begin DC-Ops team steps:**
[] - system disks removed (by onsite)
[] - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
[] - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
[] - IF DECOM: mgmt dns entries removed.
[] - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag
`cloudelastic1002.wikimedia.org`
**Steps for service owner:**
[x] - all system services confirmed offline from production use
[x] - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
[x] - remove system from all lvs/pybal active configuration
[x] - any service group puppet/hiera/dsh config removed
[x] - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
[x] - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
[x] - remove all remaining puppet references and all host entries in the puppet repo
[x] - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.
**End service owner steps / Begin DC-Ops team steps:**
[] - system disks removed (by onsite)
[] - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
[] - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
[] - IF DECOM: mgmt dns entries removed.
[] - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag
`cloudelastic1003.wikimedia.org`
**Steps for service owner:**
[x] - all system services confirmed offline from production use
[x] - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
[x] - remove system from all lvs/pybal active configuration
[x] - any service group puppet/hiera/dsh config removed
[x] - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
[x] - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
[x] - remove all remaining puppet references and all host entries in the puppet repo
[x] - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.
**End service owner steps / Begin DC-Ops team steps:**
[] - system disks removed (by onsite)
[] - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
[] - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
[] - IF DECOM: mgmt dns entries removed.
[] - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag
`cloudelastic1004.wikimedia.org`
**Steps for service owner:**
[x] - all system services confirmed offline from production use
[x] - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
[x] - remove system from all lvs/pybal active configuration
[x] - any service group puppet/hiera/dsh config removed
[x] - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
[x] - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
[x] - remove all remaining puppet references and all host entries in the puppet repo
[x] - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.
**End service owner steps / Begin DC-Ops team steps:**
[] - system disks removed (by onsite)
[] - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
[] - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
[] - IF DECOM: mgmt dns entries removed.
[] - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag