I have depooled this host to give it a kernel upgrade for T264154 (I won't repool it anymore).

This host was rebooted, and expected, never came back. The idrac also doesn't work...

Change 644408 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es1017: Remove from dbctl

https://gerrit.wikimedia.org/r/644408

Change 644408 merged by Marostegui:
[operations/puppet@production] es1017: Remove from dbctl

https://gerrit.wikimedia.org/r/644408

Mentioned in SAL (#wikimedia-operations) [2020-12-01T06:54:20Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es1017 from dbctl T268825', diff saved to https://phabricator.wikimedia.org/P13490 and previous config saved to /var/cache/conftool/dbconfig/20201201-065419-marostegui.json

Change 644675 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Decommission es1017

https://gerrit.wikimedia.org/r/644675

@Volans this host has the mgmt interface down (and most likely broken) so as expected, the boot loaders cannot be wiped, how should we proceed with those issues?

The host is off by the way, it never came back from a reboot a few days ago.

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: es1017.eqiad.wmnet

• es1017.eqiad.wmnet (FAIL)
  • Failed downtime host on Icinga (likely already removed)
  • Found physical host
  • Skipped downtime management interface on Icinga (likely already removed)
  • Unable to connect to the host, wipe of bootloaders will not be performed: Cumin execution failed (exit_code=2)
  • Failed to power off, manual intervention required: Remote IPMI for es1017.mgmt.eqiad.wmnet failed (exit=1): b''
  • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

Mentioned in SAL (#wikimedia-operations) [2020-12-02T06:54:50Z] <marostegui> Remove es1017 from tendril and zarcillo T268825

Change 644675 merged by Marostegui:
[operations/puppet@production] mariadb: Decommission es1017

https://gerrit.wikimedia.org/r/644675

@Marostegui currently the wipe of bootloaders is done from the OS, not the mgmt, so if the host is already down/broken it can't be done, but is not affected by the mgmt console not working. The step is done just to make sure the host can't come back and re-use its old configured IPs more than anything else. So if the host is already broken there is no real risk and it's ok as is. The actual disk wipe is still performed by DC-Ops separately.

Ah, excellent @Volans - thanks for clarifying that.

For the record, the homer run:

# homer asw2-c-eqiad* commit "T268825"
INFO:homer.devices:Initialized 35 devices
INFO:homer:Committing config for query asw2-c-eqiad* with message: T268825
INFO:homer:Gathering global Netbox data
INFO:homer.devices:Matched 1 device(s) for query 'asw2-c-eqiad*'
INFO:homer:Generating configuration for asw2-c-eqiad.mgmt.eqiad.wmnet
Configuration diff for asw2-c-eqiad.mgmt.eqiad.wmnet:

[edit interfaces interface-range disabled]
     member ge-3/0/13 { ... }
+    member ge-3/0/18;
     member ge-3/0/22 { ... }
[edit interfaces interface-range vlan-private1-c-eqiad]
-    member ge-3/0/18;
[edit interfaces]
-   ge-3/0/18 {
-       description "es1017:eno1 {#}";
-   }

Type "yes" to commit, "no" to abort.
> yes
INFO:homer.transports.junos:Committing the configuration on asw2-c-eqiad.mgmt.eqiad.wmnet
INFO:homer:Homer run completed successfully on 1 devices: ['asw2-c-eqiad.mgmt.eqiad.wmnet']