~~Aiming at starting the work on Wednesday June 27th, 11am local time (6pm UTC), 5h estimated maintenance.~~
EDIT: Blocked by T196030
Before maintenance window:
[x] Tentative: rack/power the routers in advance (temp location), to be able to upgrade them and apply initial config
[] Add routers to racktables (cr3-ulsfo & cr4-ulsfo)
[] Confirms valid backup or cr1/cr2 configs
[] Write down the unique serial # labeled on each fiber optic cable in each network port
[] Prepare DNS/Icinga/Smokeping/Rancid CREDIT2: new plan due to DC move
During maintenance window:[x] Rack/power the routers in temp location
[] Depool ulsfo from serving traffic via dnsx] Upgrade cr3/4
[] Downtime all ulsfo hosts in Icinga/LibreNMSConfigure cr3/4
[] Unplug all cr1Disable transit/peering links
[] Unrack cr1-ulsfox] Prepare DNS/Icinga/Smokeping/Rancid CRs
[] Move cr1 patch panel to other side of rackhttps://gerrit.wikimedia.org/r/461228
[] Rack cr3-ulsfo
[] Connect cr3 uplinks (ports in descriptions)https://gerrit.wikimedia.org/r/461233
During maintenance window:
[] Configure cr3-ulsfoDepool ulsfo from serving traffic via dns
[] Verify cr3 in healthy state (rollback decisionDowntime all ulsfo hosts in Icinga/LibreNMS (and devices linked to ulsfo hosts)
[] Unplug all cr2 linksMove cr3/4 to new racks
[] Unrack cr2-ulsfoConnect cr3<->cr4 link
[] Move cr2 patch panel to other side of rackConnect cr3/4<->asw2 link
[] Rack cr4-ulsfoReconfigure asw2 for new router
[] Connect cr4 uplinks (3/4 transports TBD) links
[] Configure/upgrade cr4-Verify all sessions are up, no alarms, prefixes exchanged, ulsfo devices reachable
[] Verify cr4 in healthy stateConnect/enable cr3/4 peering/transit
[] Merge MonitorVerify all sessions are up, no alarms, prefixes exchanged, ulsfo devices reachable
[] Merge pending CR for cr3/4s (DNS then puppet)
[] AddLibreNMS, delete cr3/4, rename cr1/2 to cr3/4 to LibreNMS(to keep history)
[] Verify monitoring if happy
[] Re-pool ulsfo
After maintenance:
[] Cleanup mentions of cr1/2 in DNS, monitoring
[] Rename neighbor's interfaces descriptions
[] Update racktables
[] Wipe/unrack cr1/cr2