Similar to what we did with all the other rows (except eqiad-C so far).
The current way the row D switches are connected to each others doesn't follow Juniper recommended cabling, and could bring instability to the row (eg. as seen with T252797#6137927).
It is also a pre-requisite to add a 3rd 10G switch (T196487).
On the other rows, the recabling went smooth (no user facing impact), except one that caused few seconds downtime across the row.
The plan (and what we did previously) is to pre-cable the missing VC links (with VC Ports disabled) then enable/disable ports in the proper order, finish by removing the obsolete cables.
If any sign of instability, "rip off the bandaid" and turn all the ports on/off at once so we're only left with the good design. While keeping the commands handy to rollback at any time.
This will happened during the DC failover, we need to:
- Schedule a 1h window with the service owners and DCops - Thursday 24th, 1pm UTC
- For each service owners decide if they can/want to depool their service out of eqiad row D (all servers impacted)
- Decide if we want to use that opportunity to do T196487 (which mean a ~30min hard down of rack D4) - yes, next day
- Decide if we want to use that opportunity to do T247881 (which mean a ~30min hard down of rack D1) - TBD
A few days before the window:
[DCops] Remove unused cables:
FPC1:1/0 (DAC 3m)
FPC8:1/0 (DAC 3m)
[Netops] Disable unused VC ports to not risk them going online at connect:
request virtual-chassis vc-port delete pic-slot 1 member 3 port 3 request virtual-chassis vc-port delete pic-slot 1 member 4 port 2 request virtual-chassis vc-port delete pic-slot 1 member 4 port 3 request virtual-chassis vc-port delete pic-slot 1 member 5 port 2 request virtual-chassis vc-port delete pic-slot 1 member 5 port 3 request virtual-chassis vc-port delete pic-slot 1 member 6 port 3 request virtual-chassis vc-port delete pic-slot 1 member 8 port 0 request virtual-chassis vc-port delete pic-slot 1 member 8 port 3
[DCops] Pre-cable:
FPC1:1/0 - FPC7:0/51 2xQSFP+-40G-SR4+MMF FIBER (MTP/MPO) 15m?
FPC3:1/3 - FPC7:0/52 7M DAC
FPC4:1/2 - FPC2:0/51 5M DAC
FPC5:1/2 - FPC7:0/53 5M DAC
FPC6:1/3 - FPC2:0/52 7M DAC
FPC8:1/0 - FPC2:0/53 2xQSFP+-40G-SR4+MMF FIBER (MTP/MPO) 15m?
[DCops/Netops] Update Netbox
[Netops] In window: turn VC-ports on/off for proper cabling:
request virtual-chassis vc-port set pic-slot 1 member 1 port 0 request virtual-chassis vc-port set pic-slot 0 member 7 port 51 request virtual-chassis vc-port delete pic-slot 1 member 1 port 1 request virtual-chassis vc-port delete pic-slot 1 member 3 port 0 ---------- request virtual-chassis vc-port set pic-slot 0 member 2 port 53 request virtual-chassis vc-port set pic-slot 1 member 8 port 0 request virtual-chassis vc-port delete pic-slot 1 member 6 port 1 request virtual-chassis vc-port delete pic-slot 1 member 8 port 1 ---------- request virtual-chassis vc-port set pic-slot 1 member 3 port 3 request virtual-chassis vc-port set pic-slot 0 member 7 port 52 request virtual-chassis vc-port delete pic-slot 1 member 3 port 2 request virtual-chassis vc-port delete pic-slot 1 member 4 port 0 request virtual-chassis vc-port set pic-slot 1 member 4 port 2 request virtual-chassis vc-port set pic-slot 0 member 2 port 51 ---------- request virtual-chassis vc-port set pic-slot 1 member 6 port 3 request virtual-chassis vc-port set pic-slot 0 member 2 port 52 request virtual-chassis vc-port delete pic-slot 1 member 5 port 1 request virtual-chassis vc-port delete pic-slot 1 member 6 port 0 request virtual-chassis vc-port set pic-slot 1 member 5 port 2 request virtual-chassis vc-port set pic-slot 0 member 7 port 53
[DCops] Remove now unused cables and update Netbox
FPC1:1/1 - FPC3:1/0
FPC3:1/2 - FPC4:1/0
FPC5:1/1 - FPC6:1/0
FPC6:1/1 - FPC8:1/1