High-level task to track the work to configure the new network devices that have been delivered to Eqiad under T367614 as part of normal refresh cycle for the Fundraising equipment.
The replacement equipment is the same as that recently installed in codfw for the upgrade there (see T371434), and largely the same approach will be taken in eqiad.
##### Hardware
In total we will be installing four new devices:
|Type|Name|Description|Replaces|
|------|-------|--------------|----------|
|SRX1600|pfw1a-eqiad|Firewall, in HA pair|pfw3a-eqiad|
|SRX1600|pfw1b-eqiad|Firewall, in HA pair|pfw3b-eqiad|
|QFX5120|fasw2-c1a-eqiad|Top-of-rack switch|fasw-c1a-eqiad|
|QFX5120|fasw2-c1b-eqiad|Top-of-rack switch|fasw-c1b-eqiad|
##### Changes
**No virtual-chassis**
The main difference in configuration is the two switches will not be configured to operate as a single logical device using Juniper virtual-chassis, but instead they will be independent switches connected with a 2x100G LAG operating as a regular trunk port. Frack servers are configured with both their interfaces in a single Linux 'bond', configured in [[ https://wiki.linuxfoundation.org/networking/bonding | active-backup ]] mode, which means we do not need to support any form of multi-chassis LAG so this will work fine.
**Fmsw connects directly to firewalls**
As the new switches are not going to be configured as a virtual-chassis we cannot connect each to //fmsw-c1-eqiad// as the old ones had been. Instead we will connect the management switch directly off the new firewall pair, and use a new //reth1// interface to act as a redundant gateway. This has the additional benefit of ensuring the management network is reachable regardless of the state of the fasw switches.
**Dual HA ports**
As we have sufficient ports two links will be configured for the firewall cluster control ports, and two will be configured for the cluster fabric ports. This ensures the firewall cluster will continue to operate if any single one of these links fails.
**25G Downlinks**
The new firewalls have two SFP28 ports, so we can use these to connect to the two new switches at 25G instead of the previous 10G.
### Migration Plan
##### Step 1: Firewall Installation & Cabling
NOTE: Ideally we would rack all the new devices at once, however there is limited room in the rack so we need to take an alternate approach, re-using space as we go.
To begin with we will just rack the new firewalls:
|Device|Ports facing|Rack U|
|--------|--------------|--------|
|pfw1a-eqiad|Front of rack|44|
|pfw1b-eqiad|Front of rack|43|
Please cable up as follows:
|Device 1|Front Port|Logical Int|Device 2|Front Port |Logical Int|Cable Type|Desc|
|----------|-------------|-----------|-----------|-------------|-----------|------------- |------|
|pfw1a-eqiad|HA 0|N/A|pfw1b-eqiad|HA 0|N/A|0.5m 1G DAC|Cluster control link #1|
|pfw1a-eqiad|HA 1|N/A|pfw1b-eqiad|HA 1|N/A|0.5m 1G DAC|Cluster control link #2|
|pfw1a-eqiad|20|xe-0/2/2|pfw1b-eqiad|20|xe-7/2/2|0.5m 10G DAC|Cluster fabric link #1|
|pfw1a-eqiad|21|xe-0/2/3|pfw1b-eqiad|21|xe-7/2/3|0.5m 10G DAC|Cluster fabric link #2|
|pfw1a-eqiad|MGMT|fxp0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Mgmt Network|
|pfw1b-eqiad|MGMT|fxp0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Mgmt Network|
|pfw1a-eqiad|CON|N/A|scs-c1-eqiad|37|N/A|RJ45 patch|Serial console access|
|pfw1b-eqiad|CON|N/A|scs-c1-eqiad|38|N/A|RJ45 patch|Serial console access|
|pfw1a-eqiad|0|ge-0/0/0|fmsw-c1-eqiad|(any free port)|N/A|RJ45 patch|Downstream connectivity to fmsw #1 (reth1)|
|pfw1b-eqiad|0|ge-7/0/0|fmsw-c1-eqiad|(any free port)|N/A|RJ45 patch|Downstream connectivity to fmsw #2 (reth1)|
##### Step 2: Migrate from pfw3-eqiad firewall pair to newly-installed pfw1-eqiad pair
(WARNING) Netops will need to configure the new firewalls after installation before we can proceed with this step.
The basic work at this step is to move the cables coming from our CR routers, and the existing fasw pair in rack C1, from the old firewalls to the new firewalls. In other words we remove the old firewalls from the traffic path and put the new ones in their place.
(IMPORTANT) This step will disrupt connectivity to all the frack network while the links are moved and ARP updates. Interruption is estimated to be in the range of a few minutes.
The following links should be moved:
|Old Device|Old Front Port|Old Logical Port|New Device|New Front Port|New Logical Port|Desc|
|-------------|------------------|-----------------|----------------|------------------|------------------|--------|
|pfw3a-eqiad|0/17|xe-0/0/17|pfw1a-eqiad|18|xe-0/2/0|Downlink to fasw-c1a-eqiad xe-0/2/0|
|pfw3b-eqiad|0/17|xe-7/0/17|pfw1b-eqiad|18|xe-7/2/0|Downlink to fasw-c1b-eqiad xe-1/2/0|
|pfw3a-eqiad|0/16|xe-0/0/16|pfw1a-eqiad|16|xe-0/1/0|Uplink to cr1-codfw xe-3/1/7|
|pfw3b-eqiad|0/16|xe-7/0/16|pfw1b-eqiad|16|xe-7/1/0|Uplink to cr2-codfw xe-3/1/7|
Due to the pre-configuration, once the cr links are moved BGP should break and re-establish using the same link IPs as had been used on the old firewalls. Similarly the sub-interfaces of //reth0// on the new firewall pair should take over as gateway for hosts connected to the existing switches.
NOTE: No sub-interface of //reth0// will be configured for vlan 1140 (frack-management1-c-eqiad). Instead at this point Netops will bundle ge-0/0/0 and ge-7/0/0 on the new firewalls (directly connected to //fmsw-c1-eqiad//) into //reth1// and configure it with the GW IP for the frack mgmt range. This preserves link redundancy and improves the topology by not making the management access dependent on the fasw.
When these steps are complete full testing should be carried out to validate that all the fundraising network hosts and services are available and working after migrating to the new firewalls.
##### Step 3: Remove old firewalls from the rack, install new switches in their place
Once things have settled down (i.e. not the same day), we can remove all remaining cables terminating on pfw3a-eqiad and pfw3b-eqiad, and remove them from the rack. We also need to clean up Netbox removing these nodes.
We can then proceed and use the newly-freed space to install the new fasw switches:
|Device|Ports facing|Rack U|
|--------|--------------|--------|
|fasw2-c1a-eqiad|Back of rack|46|
|fasw2-c1a-eqiad|Back of rack|45|
They should be cabled as follows once racked:
|Device 1|Front Port|Logical Int|Device 2|Front Port |Logical Int|Cable Type|Desc|
|----------|-------------|-----------|-----------|-------------|-----------|------------- |------|
|fasw2-c1a-eqiad|C0|em0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Management network|
|fasw2-c1b-eqiad|C0|em0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Management network|
|fasw2-c1a-eqiad|CON|N/A|scs-c1-eqiad|39|N/A|RJ45 patch|Serial console access|
|fasw2-c1b-eqiad|CON|N/A|scs-c1-eqiad|42|N/A|RJ45 patch|Serial console access|
|fasw2-c1a-eqiad|54|et-0/0/54|fasw2-c1b-eqiad|54|et-0/0/54|0.5m 100G QSFP28 DAC|Trunk between new switches LAG port 1|
|fasw2-c1a-eqiad|55|et-0/0/55|fasw2-c1b-eqiad|55|et-0/0/55|0.5m 100G QSFP28 DAC|Trunk between new switches LAG port 2|
|fasw2-c1a-eqiad|TBC|TBC|pfw1a-eqiad|17|et-0/1/1|3m 25G SFP28 DAC|Uplink from new switch to firewall|
|fasw2-c1b-eqiad|TBC|TBC|pfw1a-eqiad|17|et-7/1/1|3m 25G SFP28 DAC|Uplink from new switch to firewall|
|fasw2-c1a-eqiad|TBC|TBC|fasw-c1a-eqiad|SFP+ port 1|xe-0/2/1|10G DAC Cable|Trunk from new switch to old switches LAG port 1|
|fasw2-c1a-eqiad|TBC|TBC|fasw-c1b-eqiad| SFP+ port 1|xe-1/2/1|10G DAC Cable|Trunk from new switch to old switches LAG port 2|
NOTE: The exact ports on the new switches for the last 4 links needs to be confirmed after we have a full-port plan for what servers will connect to what switch ports after the migration. Due to the QFX5120 requirement to configure port speeds in blocks of 4 we need to determine how many 1G and 10G server connections we will have before we can plan those links.
##### Step 4: Change pfw1-eqiad reth0 member ports to re-route traffic via the new switches
(WARNING) Netops will need to configure the new switches after installation before we can proceed with this step.
At this point the new firewall is connected directly to both the old switches (both ports are part of reth0) and the new switches (the ports are not yet configured). There is also a trunk from //fasw2-c1a-eqiad// to the old switches to allow for traffic to flow from NEW_SWITCH -> OLD_SWITCH -> SERVER.
At this point the configuration on the new firewall stack pfw1-eqiad should be changed to:
* Remove xe-0/2/0 and xe-7/2/0 from logical interface reth0
* Disable xe-0/2/0 and xe-7/2/0 (connecting to old switches)
* Enable et-0/1/1 and et-7/1/1 (connecting to new switches)
* Add et-0/1/1 and et-7/1/1 to logical interface reth0
As this can be done with an atomic commit on the Juniper firewall it should have almost no impact on the flow of production traffic. The logical interface, zones, connection state will not change on the firewall, as everything still routes via the reth0 sub-interfaces with the same IPs (just the member ports have changed). At the switching layer the MAC for the GW IPs will be learnt by the new switches over the newly-enabled ports, and in turn learnt by the old switches on their 2x10G trunk to //fasw2-c1a-eqiad//. The port the old switches previously seen the GW MAC on will be shut down.
(INFORMATION) As MAC flooding happens so quickly this is not expected to impact traffic, however some small interruption, or worse a mistake or unexpected event, could impact the traffic.
##### Step 5: Migrate servers
At this point all new network components are in service, and we can begin the process of moving servers from old switches to new. Netops will pre-configure the new switches for the server connections, after which the fr-tech guys and dc-ops can move the links from old to new ports one-by-one. We will create a new task to detail these moves and the ports.
##### Step 6: Tidy up
Once all servers have been moved we can decommission the old switches, and remove the trunk from them to the new switches plus any other cables remaining.