Page MenuHomePhabricator

decommission es1017.eqiad.wmnet
Closed, ResolvedPublicRequest


This task will track the decommission-hardware of server es1017.eqiad.wmnet

With the launch of updates to the decom cookbook, the majority of these steps can be handled by the service owners directly. The DC Ops team only gets involved once the system has been fully removed from service and powered down by the decommission cookbook.


Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal.
  • - remove all remaining puppet references (include role::spare) and all host entries in the puppet repo
  • - remove ALL dns entries except the asset tag mgmt entries.
  • - Run homer to disable its switch port
  • - reassign task from service owner to DC ops team member depending on site of servee.

End service owner steps / Begin DC-Ops team steps:

  • - disable switch port / set to asset tag if host isn't being unracked / remove from switch if being unracked.
  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: switch port configuration removed from switch once system is unracked.
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

Event Timeline

Marostegui created this task.
Marostegui moved this task from Triage to Ready on the DBA board.

I have depooled this host to give it a kernel upgrade for T264154 (I won't repool it anymore).

This host was rebooted, and expected, never came back. The idrac also doesn't work...

Change 644408 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es1017: Remove from dbctl

Change 644408 merged by Marostegui:
[operations/puppet@production] es1017: Remove from dbctl

Mentioned in SAL (#wikimedia-operations) [2020-12-01T06:54:20Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es1017 from dbctl T268825', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20201201-065419-marostegui.json

Change 644675 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Decommission es1017

@Volans this host has the mgmt interface down (and most likely broken) so as expected, the boot loaders cannot be wiped, how should we proceed with those issues?

The host is off by the way, it never came back from a reboot a few days ago.

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: es1017.eqiad.wmnet

  • es1017.eqiad.wmnet (FAIL)
    • Failed downtime host on Icinga (likely already removed)
    • Found physical host
    • Skipped downtime management interface on Icinga (likely already removed)
    • Unable to connect to the host, wipe of bootloaders will not be performed: Cumin execution failed (exit_code=2)
    • Failed to power off, manual intervention required: Remote IPMI for es1017.mgmt.eqiad.wmnet failed (exit=1): b''
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

Mentioned in SAL (#wikimedia-operations) [2020-12-02T06:54:50Z] <marostegui> Remove es1017 from tendril and zarcillo T268825

Change 644675 merged by Marostegui:
[operations/puppet@production] mariadb: Decommission es1017

@Marostegui currently the wipe of bootloaders is done from the OS, not the mgmt, so if the host is already down/broken it can't be done, but is not affected by the mgmt console not working. The step is done just to make sure the host can't come back and re-use its old configured IPs more than anything else. So if the host is already broken there is no real risk and it's ok as is. The actual disk wipe is still performed by DC-Ops separately.

Ah, excellent @Volans - thanks for clarifying that.

Marostegui updated the task description. (Show Details)
Marostegui added a project: DC-Ops.
Marostegui edited projects, added ops-eqiad; removed DBA.

Ready for DC-Ops

For the record, the homer run:

# homer asw2-c-eqiad* commit "T268825"
INFO:homer.devices:Initialized 35 devices
INFO:homer:Committing config for query asw2-c-eqiad* with message: T268825
INFO:homer:Gathering global Netbox data
INFO:homer.devices:Matched 1 device(s) for query 'asw2-c-eqiad*'
INFO:homer:Generating configuration for asw2-c-eqiad.mgmt.eqiad.wmnet
Configuration diff for asw2-c-eqiad.mgmt.eqiad.wmnet:

[edit interfaces interface-range disabled]
     member ge-3/0/13 { ... }
+    member ge-3/0/18;
     member ge-3/0/22 { ... }
[edit interfaces interface-range vlan-private1-c-eqiad]
-    member ge-3/0/18;
[edit interfaces]
-   ge-3/0/18 {
-       description "es1017:eno1 {#}";
-   }

Type "yes" to commit, "no" to abort.
> yes
INFO:homer.transports.junos:Committing the configuration on asw2-c-eqiad.mgmt.eqiad.wmnet
INFO:homer:Homer run completed successfully on 1 devices: ['asw2-c-eqiad.mgmt.eqiad.wmnet']

removed from rack, updated netbox and ran the script, confirmed network ports were already removed.