The recabling should not cause any service interruption (even though it caused some few seconds downtime for a similar recabling in eqiad but none for codfw row D).
All servers in row D are listed on https://netbox.wikimedia.org/dcim/devices/?q=&rack_group_id=14&role=server&status=1
The rack D4 switch replacement will cause up to 30min downtime for the following servers:
https://netbox.wikimedia.org/dcim/devices/?q=&rack_group_id=14&rack_id=70&role=server&status=1
mc2033, 10*mw, wdqs2006, sessionstore2003, ores2008
cc
@Eevans for sessionstore2003 (T209389)
@elukey for mc2033 and mw and "to ensure we removed any mcrouter proxies from the config"
@Gehel for wdqs2006
@akosiaris for ores2008
Looking at doing it Wednesday December 19th - 4pm UTC - 10am Dallas time - 3h
1/preparations
- Rack QFX [papaul]
- Connect console [papaul]
- Connect USB drive containing Junos 14.1X53-D43.7 (present in install2002:/home/ayounsi/jinstall-qfx-5-14.1X53-D43.7-domestic-signed.tgz) [papaul]
- Pre-populate SFP-Ts [papaul]
ge-4/0/0 ge-4/0/1 ge-4/0/2 ge-4/0/3 ge-4/0/4 ge-4/0/5 ge-4/0/6 ge-4/0/7 ge-4/0/8 ge-4/0/9 ge-4/0/10 ge-4/0/11 ge-4/0/12
- Upgrade and configure VCP on QFX [arzhel]
request system software add jinstall-qfx-5-14.1X53-D43.7-domestic-signed.tgz force-host... request virtual-chassis mode fabric mixed local request virtual-chassis vc-port set pic-slot 0 port 52 local request virtual-chassis vc-port set pic-slot 0 port 53 local request system zeroize
- Get QFX serial#
- Pre run (but don't connect) VC links [papaul]
2/ recabling
To be on the safe side:
- Depool site in DNS [arzhel]
- Redirect eqsin/ulsfo caches to eqiad [arzhel]
- Downtime VC ports Icinga alert [arzhel]
- Insert uplink module to A8 (hot-insertable) [papaul]
- Enable all VC ports (except uplinks) on spines [arzhel]
- Enable VC ports on fpc8 uplink module [arzhel]
- Add: [papaul]
Links are 40G unless 10G is specified
fpc2-fpc4
fpc5-fpc7
- Confirm working [arzhel]
- Remove: [papaul]
fpc3:1/2-fpc4:1/0
fpc3:1/0-fpc1:1/1
fpc5:1/1-fpc6:1/0
fpc1:1/0-fpc8:1/0
fpc6:1/1-fpc8:1/1
- Add: [papaul]
fpc1-fpc7
fpc2-fpc6
fpc2-fpc8 (2*10G)
- Confirm working [arzhel]
- Remove fpc8:1/2-fpc7:0/50 [papaul]
- Add fpc8-fpc7 (with 2*10G) [papaul]
- Add fpc3-fpc7 [papaul]
- cleanup unused VC ports [arzhel]
3/ FPC4 replacement
- Downtime hosts in Icinga [arzhel]
- Shutdown EX [arzhel]
- Reconfigure VCP with QFX serial# [arzhel]
set virtual-chassis member 4 serial-number XXXX
- Power on QFX [papaul]
- Connect console [papaul]
- Move VC cables from EX to QFX (ports 52/53) [papaul]
- Move servers' uplinks from EX to QFX [papaul]
- verify monitoring is happy [arzhel]
- Repool site [arzhel]
- Update Netbox (rename old/new a4 switches, serial connection, etc) [papaul]
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/vcf-removing.html
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/vcf-adding-device.html
https://www.juniper.net/documentation/en_US/release-independent/junos/topics/reference/specifications/uplink-module-ex4300.html