As the new codfw is ready is configured, it's time to move the servers over and test it.
That will start on Tuesday August 8th, at 15:00 UTC / 10:00 CDT / 08:00 PST, and should takes max 4h, 6h if need to rollback.
During the window (2h):
- Deactivate interfaces on cr1/2-codfw to pfw1/2-codfw
- Advertise codfw-frack routes from pfw3-codfw
delete protocols bgp group Production export NONE delete protocols bgp group VPN export NONE set protocols bgp group Production export BGP_fundraising_export set protocols bgp group VPN export BGP_fundraising_aggregates
- Repatch servers to new switch stack
| hostname | new port | old port |
| payments2001 | ge-0/0/0 | pfw1:ge-2/0/0 |
| payments2003 | ge-0/0/1 | pfw1:ge-2/0/1 |
| pay-lvs2001 | ge-0/0/2 | pfw1:ge-2/0/2 |
| heka | ge-0/0/5 | pfw1:ge-2/0/3 |
| saiph | ge-0/0/13 | pfw1:ge-2/0/4 |
| alnilam | ge-0/0/9 | pfw1:ge-2/0/5 |
| rigel | ge-0/0/14 | pfw1:ge-2/0/6 |
| frdb2001 | ge-0/0/10 | pfw1:ge-2/0/7 |
| frbackup2001 | ge-0/0/6 | pfw1:ge-2/0/8 |
| payments2002 | ge-1/0/3 | pfw2:ge-11/0/0 |
| pay-lvs2002 | ge-1/0/4 | pfw2:ge-11/0/1 |
| mintaka | ge-1/0/11 | pfw2:ge-11/0/2 |
| alnitak | ge-1/0/12 | pfw2:ge-11/0/3 |
| bellatrix | ge-1/0/7 | pfw2:ge-11/0/4 |
| betelgeuse | ge-1/0/8 | pfw2:ge-11/0/5 |
After the migration (2h) testing:
- Verify monitoring is all green
- Verify BGP sessions are UP (pybal)
- Do failover tests (unplug each devices and core links, verify failover time/behavior)
- Verify NAT
- Verify cross DC syncs
Rollback decision
Cleanup
- cr1/2
# That's mr1-codfw (unrelated) delete policy-options prefix-list fundraising-codfw4 208.80.153.196/32 # Not needed after migration (old pfw-codfw lo0) delete policy-options prefix-list fundraising-codfw4 208.80.153.195/32 # Old BGP neighbor IPs delete protocols bgp group fundraising neighbor 208.80.153.215 delete protocols bgp group fundraising neighbor 208.80.153.217 # Multipath not needed delete protocols bgp group fundraising multipath # Static routes only needed during transition delete routing-options static route 208.80.153.197/32
Set previous interfaces to pfw-codfw as disabled
- pfw-eqiad
delete firewall family inet filter loopback4 term allow_codfw from source-address 208.80.153.195/32
Remove IPsec/BGP to old pfw-codfw
- Remove dns entries
- Remove rancid config
- Remove from Icinga
- Remove from LibreNMS
- Remove from torrus
Unrack final part of rack elevation, back to T169643