cookbooks.sre.hosts.decommission executed by ayounsi@cumin1002 for hosts: sretest1004.eqiad.wmnet

• sretest1004.eqiad.wmnet (FAIL)
  • Host not found on Icinga, unable to downtime it
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
  • Powered off
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Host steps raised exception: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

Change #1091711 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/dns@master] Remove v6 include for e8/f8 uplinks

https://gerrit.wikimedia.org/r/1091711

I've tidied up netbox for these now.

I left the ports enabled on the ssw side with the IPs present, as we can't disable them there and keep the IPs attached. Easier to do that and re-use them for the replacement Junipers than to patch the dns repo to remove the /64 reverse include statements. I've also kept the gateway IPs in netbox, renamed to "irb-XXX" for the Junipers, but currently unattached to any interface. Again so we can reuse the subnets and avoid patching the dns repo in the short term.

All good for them to be removed from racks and cables removed. Please delete the cables/connections from Netbox to match what is done on site. Thanks.

In T380050#10613653, @cmooney wrote:

Please delete the cables/connections from Netbox to match what is done on site.

For the record I deleted the cables going from Dells to the Spines in Netbox, and added new ones from the Junipers to the same Spine ports. In reality all we need to do is move the cable from the Dells to the Junipers to match.

Re: https://gerrit.wikimedia.org/r/c/operations/dns/+/1091711/comments/5e6962e8_b88980ce - Do the IPs need to be deleted from netbox?

Change #1091711 abandoned by Ayounsi:

[operations/dns@master] Remove v6 include for e8/f8 uplinks

Reason:

reused for https://phabricator.wikimedia.org/T382017

https://gerrit.wikimedia.org/r/1091711

@VRiley-WMF These are still listed as in rack in netbox and decom status please update and fix

In T380050#10654652, @BCornwall wrote:

Re: https://gerrit.wikimedia.org/r/c/operations/dns/+/1091711/comments/5e6962e8_b88980ce - Do the IPs need to be deleted from netbox?

Yeah I'll tidy that up shortly. We re-used the original IPs for the connections to the replacement Juniper switches that went in.

But that was something of a mistake. As the Junipers are part of the EVPN cluster they should have only had IPv4 addresses on the links (everything else is tunneled in VXLAN) plus those link IPs should really come from a dedicated block. So while things are working now they have IPv6 addresses that aren't needed, and the v4 addresses are for the incorrect block.

Everything is working fine, but for the sake of avoiding confusion I'll renumber things this week so it's clear, and re-submit the above patch.