High-level task to track the work to configure the new network devices that have been delivered to Eqiad under T367614 as part of normal refresh cycle for the Fundraising equipment.
The replacement equipment is the same as that recently installed in codfw for the upgrade there (see T371434), and largely the same approach will be taken in eqiad.
##### Hardware
In total we will be installing four new devices:
|Type|Name|Description|Replaces|
|------|-------|--------------|----------|
|SRX1600|pfw1a-eqiad|Firewall, in HA pair|pfw3a-eqiad|
|SRX1600|pfw1b-eqiad|Firewall, in HA pair|pfw3b-eqiad|
|QFX5120|fasw2-c1a-eqiad|Top-of-rack switch|fasw-c1a-eqiad|
|QFX5120|fasw2-c1b-eqiad|Top-of-rack switch|fasw-c1b-eqiad|
##### Changes
**No virtual-chassis**
The main difference in configuration is the two switches will not be configured to operate as a single logical device using Juniper virtual-chassis, but instead they will be independent switches connected with a 2x100G LAG operating as a regular trunk port. Frack servers are configured with both their interfaces in a single Linux 'bond', configured in [[ https://wiki.linuxfoundation.org/networking/bonding | active-backup ]] mode, which means we do not need to support any form of multi-chassis LAG so this will work fine.
**Fmsw connects directly to firewalls**
As the new switches are not going to be configured as a virtual-chassis we cannot connect each to //fmsw-c1-eqiad// as the old ones had been. Instead we will connect the management switch directly off the new firewall pair, and use a new //reth1// interface to act as a redundant gateway. This has the additional benefit of ensuring the management network is reachable regardless of the state of the fasw switches.
**Dual HA ports**
As we have sufficient ports two links will be configured for the firewall cluster control ports, and two will be configured for the cluster fabric ports. This ensures the firewall cluster will continue to operate if any single one of these links fails.
**25G Downlinks**
The new firewalls have two SFP28 ports, so we can use these to connect to the two new switches at 25G instead of the previous 10G.
### Migration Plan
##### Step 1: Rack new devices (complete)
NOTE: This was originally planned in more steps over more days, but John was able to move the WMF prod management switch in the rack which means we can install all the new gear in parallel with the old equipment.
First step is to rack the new equipment as follows:
|Device|Ports facing|Rack U|
|--------|--------------|--------|
|pfw1a-eqiad|Front of rack|42|
|pfw1b-eqiad|Front of rack|41|
|fasw2-c1a-eqiad|Back of rack|40|
|fasw2-c1b-eqiad|Back of rack|39|
##### Step 2: Initial cabling for the new devices
Next we do all the new cabling for the new devices, but without interfering with any of the old equipment or links
|Device 1|Front Port|Logical Int|Device 2|Front Port |Logical Int|Cable Type|Desc|
|----------|-------------|-----------|-----------|-------------|-----------|------------- |------|
|pfw1a-eqiad|HA 0|N/A|pfw1b-eqiad|HA 0|N/A|0.5m 1G DAC|Cluster control link #1|
|pfw1a-eqiad|HA 1|N/A|pfw1b-eqiad|HA 1|N/A|0.5m 1G DAC|Cluster control link #2|
|pfw1a-eqiad|20|xe-0/2/2|pfw1b-eqiad|20|xe-7/2/2|0.5m 10G DAC|Cluster fabric link #1|
|pfw1a-eqiad|21|xe-0/2/3|pfw1b-eqiad|21|xe-7/2/3|0.5m 10G DAC|Cluster fabric link #2|
|pfw1a-eqiad|MGMT|fxp0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Mgmt Network|
|pfw1b-eqiad|MGMT|fxp0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Mgmt Network|
|pfw1a-eqiad|CON|N/A|scs-c1-eqiad|37|N/A|RJ45 patch|Serial console access|
|pfw1b-eqiad|CON|N/A|scs-c1-eqiad|38|N/A|RJ45 patch|Serial console access|
|pfw1a-eqiad|0|ge-0/0/0|fmsw-c1-eqiad|(any free port)|N/A|RJ45 patch|Downstream connectivity to fmsw #1 (reth1)|
|pfw1b-eqiad|0|ge-7/0/0|fmsw-c1-eqiad|(any free port)|N/A|RJ45 patch|Downstream connectivity to fmsw #2 (reth1)|
|fasw2-c1a-eqiad|C0|em0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Management network|
|fasw2-c1b-eqiad|C0|em0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Management network|
|fasw2-c1a-eqiad|CON|N/A|scs-c1-eqiad|39|N/A|RJ45 patch|Serial console access|
|fasw2-c1b-eqiad|CON|N/A|scs-c1-eqiad|42|N/A|RJ45 patch|Serial console access|
|fasw2-c1a-eqiad|54|et-0/0/54|fasw2-c1b-eqiad|54|et-0/0/54|0.5m 100G QSFP28 DAC|Trunk between new switches LAG port 1|
|fasw2-c1a-eqiad|55|et-0/0/55|fasw2-c1b-eqiad|55|et-0/0/55|0.5m 100G QSFP28 DAC|Trunk between new switches LAG port 2|
|fasw2-c1a-eqiad|47|et-0/0/47|pfw1a-eqiad|17|et-0/1/1|3m 25G SFP28 DAC|Uplink from new switch to firewall|
|fasw2-c1b-eqiad|47|et-0/0/47|pfw1a-eqiad|17|et-7/1/1|3m 25G SFP28 DAC|Uplink from new switch to firewall|
|fasw2-c1a-eqiad|43|xe-0/0/43|fasw-c1a-eqiad|SFP+ port 1|xe-0/2/1|10G DAC Cable|Trunk from new switch to old switches LAG port 1|
|fasw2-c1a-eqiad|43|xe-0/0/43|fasw-c1b-eqiad| SFP+ port 1|xe-1/2/1|10G DAC Cable|Trunk from new switch to old switches LAG port 2|
(WARNING) Before we proceed with the next step Netops need to complete the provisioning and configuration of the new devices ready for the migration.
##### Step 3: Cut-over from old firewalls/switches to new pair
The basic work at this step is to move the cables coming from our CR routers, from the old firewalls to the new firewalls. In other words we remove the old firewalls from the traffic path and put the new ones in their place. The new firewalls are connected to the new switches in advance, which in turn are connected to the old switches which makes the servers reachable.
(IMPORTANT) This step will disrupt connectivity to all the frack network while the links are moved and ARP updates. Interruption is estimated to be in the range of a few minutes.
The following links should be moved:
|Old Device|Old Front Port|Old Logical Port|New Device|New Front Port|New Logical Port|Desc|
|-------------|------------------|-----------------|----------------|------------------|------------------|--------|
|pfw3a-eqiad|0/17|xe-0/0/17|pfw1a-eqiad|18|xe-0/2/0|Downlink to fasw-c1a-eqiad xe-0/2/0|
|pfw3b-eqiad|0/17|xe-7/0/17|pfw1b-eqiad|18|xe-7/2/0|Downlink to fasw-c1b-eqiad xe-1/2/0|
Due to the pre-configuration, once the cr links are moved BGP should break and re-establish using the same link IPs as had been used on the old firewalls. Similarly the sub-interfaces of //reth0// on the new firewall pair should take over as gateway for hosts connected to the existing switches.
NOTE: Simultaneously netops will disable xe-0/2/0 and xe-1/2/0 on old switch-stack fasw-c-eqiad. This will disable the outbound flow of traffic from the old switches via the old firewalls. Instead this traffic will re-route via the trunk from old switches to new, and out via the new firewalls.
When these steps are complete full testing should be carried out to validate that all the fundraising network hosts and services are available and working after migrating to the new firewalls.
##### Step 5: Migrate servers
At this point all new network components are in service, and we can begin the process of moving servers from old switches to new. Netops will pre-configure the new switches for the server connections, after which the fr-tech guys and dc-ops can move the links from old to new ports one-by-one. We will create a new task to detail these moves and the ports.
##### Step 6: Tidy up
Once all servers have been moved we can decommission the old switches, and remove the trunk from them to the new switches plus any other cables remaining.