High-level task to track the work to configure the new network devices that have been delivered to Eqiad under T367614 as part of normal refresh cycle for the Fundraising equipment.
The replacement equipment is the same as that recently installed in codfw for the upgrade there (see T371434), and largely the same approach will be taken in eqiad.
##### Hardware
In total we will be installing four new devices:
|Type|Name|Description|Replaces|
|------|-------|--------------|----------|
|SRX1600|pfw1a-eqiad|Firewall, in HA pair|pfw3a-eqiad|
|SRX1600|pfw1b-eqiad|Firewall, in HA pair|pfw3b-eqiad|
|QFX5120|fasw2-c1a-eqiad|Top-of-rack switch|fasw-c1a-eqiad|
|QFX5120|fasw2-c1b-eqiad|Top-of-rack switch|fasw-c1b-eqiad|
##### Changes
**No virtual-chassis**
The main difference in configuration is the two switches will not be configured to operate as a single logical device using Juniper virtual-chassis, but instead they will be independent switches connected with a 2x100G LAG operating as a regular trunk port. Frack servers are configured with both their interfaces in a single Linux 'bond', configured in [[ https://wiki.linuxfoundation.org/networking/bonding | active-backup ]] mode, which means we do not need to support any form of multi-chassis LAG so this will work fine.
**Fmsw connects directly to firewalls**
As the new switches are not going to be configured as a virtual-chassis we cannot connect each to //fmsw-c1-eqiad// as the old ones had been. Instead we will connect the management switch directly off the new firewall pair, and use a new //reth1// interface to act as a redundant gateway. This has the additional benefit of ensuring the management network is reachable regardless of the state of the fasw switches.
**Dual HA ports**
As we have sufficient ports two links will be configured for the firewall cluster control ports, and two will be configured for the cluster fabric ports. This ensures the firewall cluster will continue to operate if any single one of these links fails.
**25G Downlinks**
The new firewalls have two SFP28 ports, so we can use these to connect to the two new switches at 25G instead of the previous 10G.
### Migration Plan
##### Step 1: Firewall Installation & CablingRack new devices (complete)
NOTE: Ideally we would rack all the new devices at once, however there is limited room in the rack so we need to take an alternate approachThis was originally planned in more steps over more days, re-using space as we gobut John was able to move the WMF prod management switch in the rack which means we can install all the new gear in parallel with the old equipment.
To begin with we will justFirst step is to rack the new firewalls:equipment as follows:
|Device|Ports facing|Rack U|
|--------|--------------|--------|
|pfw1a-eqiad|Front of rack|42|
|pfw1b-eqiad|Front of rack|41|
|fasw2-c1a-eqiad|Back of rack|40|
|fasw2-c1b-eqiad|Back of rack|39|
##### Step 2: Initial cabling for the new devices
PleaseNext we do all the new cable up as follows:ing for the new devices, but without interfering with any of the old equipment or links
|Device 1|Front Port|Logical Int|Device 2|Front Port |Logical Int|Cable Type|Desc|
|----------|-------------|-----------|-----------|-------------|-----------|------------- |------|
|pfw1a-eqiad|HA 0|N/A|pfw1b-eqiad|HA 0|N/A|0.5m 1G DAC|Cluster control link #1|
|pfw1a-eqiad|HA 1|N/A|pfw1b-eqiad|HA 1|N/A|0.5m 1G DAC|Cluster control link #2|
|pfw1a-eqiad|20|xe-0/2/2|pfw1b-eqiad|20|xe-7/2/2|0.5m 10G DAC|Cluster fabric link #1|
|pfw1a-eqiad|21|xe-0/2/3|pfw1b-eqiad|21|xe-7/2/3|0.5m 10G DAC|Cluster fabric link #2|
|pfw1a-eqiad|MGMT|fxp0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Mgmt Network|
|pfw1b-eqiad|MGMT|fxp0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Mgmt Network|
|pfw1a-eqiad|CON|N/A|scs-c1-eqiad|37|N/A|RJ45 patch|Serial console access|
|pfw1b-eqiad|CON|N/A|scs-c1-eqiad|38|N/A|RJ45 patch|Serial console access|
|pfw1a-eqiad|0|ge-0/0/0|fmsw-c1-eqiad|(any free port)|N/A|RJ45 patch|Downstream connectivity to fmsw #1 (reth1)|
|pfw1b-eqiad|0|ge-7/0/0|fmsw-c1-eqiad|(any free port)|N/A|RJ45 patch|Downstream connectivity to fmsw #2 (reth1)|
##### Step 2: Migrate from pfw3-eqiad firewall pair to newly-installed pfw1-eqiad pair
(WARNING) Netops will need to configure the new firewalls after installation before we can proceed with this step.
The basic work at this step is to move the cables coming from our CR routers, and the existing fasw pair in rack C1, from the old firewalls to the new firewalls. In other words we remove the old firewalls from the traffic path and put the new ones in their place.
(IMPORTANT) This step will disrupt connectivity to all the frack network while the links are moved and ARP updates. Interruption is estimated to be in the range of a few minutes.
The following links should be moved:
|Old Device|Old Front Port|Old Logical Port|New Device|New Front Port|New Logical Port|Desc|
|-------------|------------------|-----------------|----------------|------------------|------------------|--------|
|pfw3a-eqiad|0/17|xe-0/0/17|pfw1a-eqiad|18|xe-0/2/0|Downlink to fasw-c1a-eqiad xe-0/2/0|
|pfw3b-eqiad|0/17|xe-7/0/17|pfw1b-eqiad|18|xe-7/2/0|Downlink to fasw-c1b-eqiad xe-1/2/0|
|pfw3a-eqiad|0/16|xe-0/0/16|pfw1a-eqiad|16|xe-0/1/0|Uplink to cr1-codfw xe-3/1/7|
|pfw3b-eqiad|0/16|xe-7/0/16|pfw1b-eqiad|16|xe-7/1/0|Uplink to cr2-codfw xe-3/1/7|
Due to the pre-configuration, once the cr links are moved BGP should break and re-establish using the same link IPs as had been used on the old firewalls. Similarly the sub-interfaces of //reth0// on the new firewall pair should take over as gateway for hosts connected to the existing switches.
NOTE: No sub-interface of //reth0// will be configured for vlan 1140 (frack-management1-c-eqiad). Instead at this point Netops will bundle ge-0/0/0 and ge-7/0/0 on the new firewalls (directly connected to //fmsw-c1-eqiad//) into //reth1// and configure it with the GW IP for the frack mgmt range. This preserves link redundancy and improves the topology by not making the management access dependent on the fasw.
When these steps are complete full testing should be carried out to validate that all the fundraising network hosts and services are available and working after migrating to the new firewalls.
##### Step 3: Remove old firewalls from the rack, install new switches in their place
Once things have settled down (i.e. not the same day), we can remove all remaining cables terminating on pfw3a-eqiad and pfw3b-eqiad, and remove them from the rack. We also need to clean up Netbox removing these nodes.
We can then proceed and use the newly-freed space to install the new fasw switches:
|Device|Ports facing|Rack U|
|--------|--------------|--------|
|fasw2-c1a-eqiad|Back of rack|46|
|fasw2-c1b-eqiad|Back of rack|45|
They should be cabled as follows once racked:
|Device 1|Front Port|Logical Int|Device 2|Front Port |Logical Int|Cable Type|Desc|
|----------|-------------|-----------|-----------|-------------|-----------|------------- |------|
|fasw2-c1a-eqiad|C0|em0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Management network|
|fasw2-c1b-eqiad|C0|em0|msw-c1-eiqad|(any free port)|N/A|RJ45 patch|WMF Management network|
|fasw2-c1a-eqiad|CON|N/A|scs-c1-eqiad|39|N/A|RJ45 patch|Serial console access|
|fasw2-c1b-eqiad|CON|N/A|scs-c1-eqiad|42|N/A|RJ45 patch|Serial console access|
|fasw2-c1a-eqiad|54|et-0/0/54|fasw2-c1b-eqiad|54|et-0/0/54|0.5m 100G QSFP28 DAC|Trunk between new switches LAG port 1|
|fasw2-c1a-eqiad|55|et-0/0/55|fasw2-c1b-eqiad|55|et-0/0/55|0.5m 100G QSFP28 DAC|Trunk between new switches LAG port 2|
|fasw2-c1a-eqiad|47|et-0/0/47|pfw1a-eqiad|17|et-0/1/1|3m 25G SFP28 DAC|Uplink from new switch to firewall|
|fasw2-c1b-eqiad|47|et-0/0/47|pfw1a-eqiad|17|et-7/1/1|3m 25G SFP28 DAC|Uplink from new switch to firewall|
|fasw2-c1a-eqiad|43|xe-0/0/43|fasw-c1a-eqiad|SFP+ port 1|xe-0/2/1|10G DAC Cable|Trunk from new switch to old switches LAG port 1|
|fasw2-c1a-eqiad|43|xe-0/0/43|fasw-c1b-eqiad| SFP+ port 1|xe-1/2/1|10G DAC Cable|Trunk from new switch to old switches LAG port 2|
NOTE: The exact ports on the new switches for the last 4 links needs to be confirmed after we have a full-port plan for what servers will connect to what switch ports after the migration. Due to the QFX5120 requirement to configure port speeds in blocks of 4 we need to determine how many 1G and 10G server connections we will have before we can plan those links.
##### Step 4: Change pfw1-eqiad reth0 member ports to re-route traffic via the new switches(WARNING) Before we proceed with the next step Netops need to complete the provisioning and configuration of the new devices ready for the migration.
(WARNING) Netops will need to configure the new ##### Step 3: Cut-over from old firewalls/switches after installation before we can proceed with this step.to new pair
At this point the new firewall is connected directly to both the old switches (both ports are part of reth0) and the new switches (the ports are not yet configured)The basic work at this step is to move the cables coming from our CR routers, from the old firewalls to the new firewalls. In other words we remove the old firewalls from the traffic path and put the new ones in their place. The new firewalls are connected to the
(IMPORTANT) This step will disrupt connectivity to all the frack network while the links are moved and ARP updates. There is also a trunk from //fasw2-c1a-eqiad// to the old switches to allow for traffic to flow from NEW_SWITCH -> OLD_SWITCH -> SERVERInterruption is estimated to be in the range of a few minutes.
At this point the configuration on the new firewall stack pfw1-eqiadThe following links should be changed to:moved:
* Remove xe-0/2/0 and xe-7/2/0 from logical interface reth0|Old Device|Old Front Port|Old Logical Port|New Device|New Front Port|New Logical Port|Desc|
* Disable xe-0/2/0 and xe-7/2/0 (connecting to old switches)|-------------|------------------|-----------------|----------------|------------------|------------------|--------|
* Enable et-0/1/1 and et-7/1/1 (connecting|pfw3a-eqiad|0/17|xe-0/0/17|pfw1a-eqiad|18|xe-0/2/0|Downlink to new switches)fasw-c1a-eqiad xe-0/2/0|
* Add et-0/1/1 and et-7/1/1 to logical interface reth0|pfw3b-eqiad|0/17|xe-7/0/17|pfw1b-eqiad|18|xe-7/2/0|Downlink to fasw-c1b-eqiad xe-1/2/0|
As this can be done with an atomic commit on the Juniper firewall it should have almost no impact on the flow of production traffic. The logical interface, zones, connection state will not change on the firewall, as everything still routes via the reth0 sub-interfaces with the same IPs (just the member ports have changed). At the switching layer the MAC for the GW IPs will be learnt by the new switches over the newly-enabled portsDue to the pre-configuration, and in turn learnt by the old switches ononce the cr links are moved BGP should break and re-establish using their 2x10G trunk to //fasw2-c1a-eqiad// same link IPs as had been used on the old firewalls. The port the old switches previously seen the GW MAC on will be shut downSimilarly the sub-interfaces of //reth0// on the new firewall pair should take over as gateway for hosts connected to the existing switches.
(NOTE) As MAC flooding happens so quickly this is not expected to impact traffic, however some small interruption, or worse a mistake or unexpected eventNOTE: Simultaneously netops will disable xe-0/2/0 and xe-1/2/0 on old switch-stack fasw-c-eqiad. This will disable the outbound flow of traffic from the old switches via the old firewalls. Instead this traffic will re-route via the trunk from old switches to new, could impactand out via the trafficnew firewalls.
When these steps are complete full testing should be carried out to validate that all the fundraising network hosts and services are available and working after migrating to the new firewalls.
##### Step 5: Migrate servers
At this point all new network components are in service, and we can begin the process of moving servers from old switches to new. Netops will pre-configure the new switches for the server connections, after which the fr-tech guys and dc-ops can move the links from old to new ports one-by-one. We will create a new task to detail these moves and the ports.
##### Step 6: Tidy up
Once all servers have been moved we can decommission the old switches, and remove the trunk from them to the new switches plus any other cables remaining.