Page MenuHomePhabricator

Decom asw-b-codfw switch stack
Closed, ResolvedPublic

Description

Now that all servers have been moved from the old asw-bX-codfw devices in row B to lsw1-bX-codfw independent switches we can now begin the work of decommissioning the old switches.

I believe these are the steps we need to complete:

  • Disable asw-b-codfw <-> ssw1-a1-codfw link [netops]
  • Disable asw-b-codfw <-> ssw1-a8-codfw link [netops]
  • Set to decommissioning status in netbox and remove from Homer [netops]
  • Remove from monitoring (LibreNMS/Rancid/Icinga) [netops]
  • Remove asw-b-codfw <-> ssw1-a1-codfw cable and optics [dc-ops]
  • Remove asw-b-codfw <-> ssw1-a8-codfw cable [dc-ops]
  • Disconnect mgmt ports and console ports [dc-ops]
  • Delete mgmt IPs in Netbox
  • Update / check all console server connections are correct in Netbox [dc-ops]
  • Power down and unrack asw-b-codfw members [dc-ops]
  • Zeroize devices over local serial [dc-ops?]
  • Update device status in Netbox [dc-ops]

@Papaul are you ok to zeroize the devices? I had planned to do this through the opengear but if you're ok to do it the same as with row A boxes that's probably the easiest.

Event Timeline

cmooney triaged this task as Medium priority.Mar 22 2024, 1:33 PM
cmooney created this task.

@cmooney what works for you works for me as well

Papaul updated the task description. (Show Details)

Change #1013545 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Remove config for ESI-LAG between codfw spines facing asw-b-codfw

https://gerrit.wikimedia.org/r/1013545

Change #1013545 merged by jenkins-bot:

[operations/homer/public@master] Remove config for ESI-LAG between codfw spines facing asw-b-codfw

https://gerrit.wikimedia.org/r/1013545

Icinga downtime and Alertmanager silence (ID=79f10d11-133e-477b-be4d-b326d7e4bcf9) set by cmooney@cumin1002 for 4:00:00 on 1 host(s) and their services with reason: prepping to decom switch stack

asw-b-codfw

Mentioned in SAL (#wikimedia-operations) [2024-03-22T14:11:53Z] <topranks> disabling LAG from asw-b-codfw to ssw-aX-codfw T360776

I've decommed the link from asw-b-codfw to the two Spines now and removed from Netbox. DC-Ops you can proceed and remove the cabling any time, for reference:

image.png (427×440 px, 29 KB)

image.png (427×440 px, 29 KB)

Change #1013557 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Remove entries for asw-b-codfw switch stack

https://gerrit.wikimedia.org/r/1013557

Mentioned in SAL (#wikimedia-operations) [2024-03-22T14:52:44Z] <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Remove asw-b-codfw from synced hiera data - cmooney@cumin1002 - T360776"

Mentioned in SAL (#wikimedia-operations) [2024-03-22T15:04:38Z] <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Remove asw-b-codfw from synced hiera data - cmooney@cumin1002 - T360776"

Change #1013557 merged by Cathal Mooney:

[operations/puppet@production] Remove entries for asw-b-codfw switch stack

https://gerrit.wikimedia.org/r/1013557

Papaul claimed this task.
Papaul updated the task description. (Show Details)