Just by reading the task, the first thing we will need before moving the firewall will be a SFP Transceiver 1000Base-LX, 1310nm SMF SingleMode Fiber Optic Module, for the HA port. I I will check if we do have any on site.

Please see below for the migration diagram.

Thanks @Papaul for the info. For the most part the diagram looks ok, a few questions/notes below. I've included a few improvements I think we can possibly tackle as part of this work, which brings the codfw setup more in line with eqiad.

The new switches should be called fasw1-f5a and fasw1-f5b, as they are the first 'fasw' to be in that rack.

I think we have two cluster control / HA ports on each unit?

• em0 and em1 as reported by the box, labelled HA0 and HA1 on the front?
• We should use both and yes they should be 1000BaseLX

I think it might make sense for us to take this opportunity to double-up the cluster fabric-links?

• Matching the setup in eqiad
• i.e. there we have front ports 20 and 21 (xe-0/2/2, xe-0/2/3) connected to the same on the second unit (xe-7/2/2, xe-7/2/3)
• This gives us extra redundancy over using a single link for this
• Should be 10G-Base-LR optics either side of both

We should connect the new switches at 25G matching eqiad, as it's worth it for the price of the optics

• So front port 16 on each unit (et-0/1/0 and et-7/1/0) should be used for the links to fasw
• This can be a 25G DAC from pfw1b-codfw to fasw1-f5b
• It should be 25G-Base-LR from pfw1a-codfw to fasw1-f5a

If possible we should connect fmsw1-f5 directly to the firewalls rather than off the local fasw's in the rack

• This keeps mgmt online if the main switches fail
• Matches the setup in eqiad - where we have the fmsw connected to ge-0/0/0 and ge-0/0/7 bundled into reth1
• However this would require a copper connection between racks, not sure if the distance here will make that impossible.
• If distance prevents it we should keep the current setup with fmsw connected to fasw's

But yep overall looks good.

@cmooney please see answers and comments below.

In T401297#11067948, @cmooney wrote:

Thanks @Papaul for the info. For the most part the diagram looks ok, a few questions/notes below. I've included a few improvements I think we can possibly tackle as part of this work, which brings the codfw setup more in line with eqiad.

The new switches should be called fasw1-f5a and fasw1-f5b, as they are the first 'fasw' to be in that rack.

I think we have two cluster control / HA ports on each unit?

• em0 and em1 as reported by the box, labelled HA0 and HA1 on the front?
• We should use both and yes they should be 1000BaseLX

I think it might make sense for us to take this opportunity to double-up the cluster fabric-links?

• Matching the setup in eqiad
• i.e. there we have front ports 20 and 21 (xe-0/2/2, xe-0/2/3) connected to the same on the second unit (xe-7/2/2, xe-7/2/3)
• This gives us extra redundancy over using a single link for this
• Should be 10G-Base-LR optics either side of both

We should connect the new switches at 25G matching eqiad, as it's worth it for the price of the optics

• So front port 16 on each unit (et-0/1/0 and et-7/1/0) should be used for the links to fasw
• This can be a 25G DAC from pfw1b-codfw to fasw1-f5b
• It should be 25G-Base-LR from pfw1a-codfw to fasw1-f5a

If possible we should connect fmsw1-f5 directly to the firewalls rather than off the local fasw's in the rack

• This keeps mgmt online if the main switches fail
• Matches the setup in eqiad - where we have the fmsw connected to ge-0/0/0 and ge-0/0/7 bundled into reth1
• However this would require a copper connection between racks, not sure if the distance here will make that impossible.
• If distance prevents it we should keep the current setup with fmsw connected to fasw's

But yep overall looks good.

"The new switches should be called fasw1-f5a and fasw1-f5b, as they are the first 'fasw' to be in that rack." Noted can be done

"I think we have two cluster control / HA ports on each unit?
em0 and em1 as reported by the box, labelled HA0 and HA1 on the front?
We should use both and yes they should be 1000BaseLX"

Yes in indeed we do have 2 HA ports. the question is do we really need redundancy and is it necessary to have both port connected? Let us keep in mind the fact that in codfw we are not doing cross-rack connections but cross-cage connections. So using more cross-connections on the patch panel where it is not necessary will put us in a situation where in the future when we need more cross-connections we will not have any. So let us think carefully and better plan on how we are utilization those cross-connections so we don;t have any issue in the future.

"I think it might make sense for us to take this opportunity to double-up the cluster fabric-links?"
Save answer as my above comments on using the cross-connections . i don't think this is necessary

"We should connect the new switches at 25G matching eqiad, as it's worth it for the price of the optics"
Yes this can be done.

"If possible we should connect fmsw1-f5 directly to the firewalls rather than off the local fasw's in the rack"
Save answer as my above comments on using the cross-connections . i don't think this is necessary

Please let me know if you have any question. Thank you

"I think we have two cluster control / HA ports on each unit?
em0 and em1 as reported by the box, labelled HA0 and HA1 on the front?
We should use both and yes they should be 1000BaseLX"

Yes in indeed we do have 2 HA ports. the question is do we really need redundancy and is it necessary to have both port connected? Let us keep in mind the fact that in codfw we are not doing cross-rack connections but cross-cage connections. So using more cross-connections on the patch panel where it is not necessary will put us in a situation where in the future when we need more cross-connections we will not have any. So let us think carefully and better plan on how we are utilization those cross-connections so we don;t have any issue in the future.

I definitely think it's worth connecting both. Tbh I think it's more important when the two units are not beside each other in a rack, and thus there is more chance of a problem with one or other of the links.

"I think it might make sense for us to take this opportunity to double-up the cluster fabric-links?"
Save answer as my above comments on using the cross-connections . i don't think this is necessary

For such a sensitive part of our infra I think we would be extremely foolish not add this redundancy. If we need to order more cross connects we can talk to Willy CyrusOne.

"If possible we should connect fmsw1-f5 directly to the firewalls rather than off the local fasw's in the rack"
Save answer as my above comments on using the cross-connections . i don't think this is necessary

Ok yeah I didn't properly think through that we are going via the fibre patch panels provided by the facility. Agreed it's probably not worth burning extra cross-connects for this rare situation. So let's keep the msw connections as they are, coming off the switch. We can reconfigue eqiad to match this when we do the same job there.

Thanks.

I Still strongly disagree we need redundancy on the HA port. The reason being that if the port goes down, this will not have any impact. On the other hand, I can accept the redundancy on the fabric links. Open for discussion.

In T401297#11068877, @Papaul wrote:

I Still strongly disagree we need redundancy on the HA port. The reason being that if the port goes down, this will not have any impact. On the other hand, I can accept the redundancy on the fabric links. Open for discussion.

As I understand the secondary unit will go into disabled state if this happens? What if the CR uplink from the primary node is also down? You're probably more familiar here than me so if I'm wrong please let me know. But in general we want as much redundancy as we can, we've bought expensive firewalls with dual HA ports, seems silly not to enable it for the cost of the optics / xconnect.

From a quick look it does seem best to have two control links for proper redundancy.

But I suggest that we do a test. In a maintenance window, unplug the control link and monitor what happens on both nodes, see how much we would need to push it to have a service interruption. @Papaul is that something you could do ?

@ayounsi I will have to work with fundraising to see when it will be best for us to do so. Thanks

@Papaul just FYI the diagram you did is now not accessible for other users for some reason

I talked to @Jgreen on IRC about the schedule, there is a maintenance window during from September 22nd to the 26th so this will be a best time for the migration. @Jgreen I can start this on September the 22nd at 10am CT. Thank you

Tested all the cross cage links (7) only 2 links are not coming up. I will do more testing tomorrow.

We tested the last 2 cross cage links for the frack migration and all is working now. We are ready for the move on the 22nd.

Icinga downtime and Alertmanager silence (ID=7b0c0458-499c-4287-8c6f-8f66dccdba91) set by pt1979@cumin2002 for 2:00:00 on 1 host(s) and their services with reason: pfw1-codfw relocation

pfw1-codfw

Icinga downtime and Alertmanager silence (ID=469fffa6-5667-4b97-b402-ebd2aefae808) set by pt1979@cumin2002 for 2:00:00 on 2 host(s) and their services with reason: pfw1-codfw relocation

fasw2-c8a-codfw,fasw2-c8b-codfw

We move pfw1b-codfw today from rack C8 in DH7 to rack F5 in DH5 and all is back up online. Before the move we did some testing
1- Disconnect second HA link from both the firewalls
All was good
2- Disconnect first and second HA links, lost connectivity to node 0 (unknown state) and node 1 pass to ineligible state after some minutes about 1 minute node 1 went into disabled state but we still have connectivity from eqiad to codfw
3- Connect back HA link 1 node 0 came back online and node 1 automatically reboot and went into hold state after about a minute it went into the initial state secondary node

Redundancy group: 1 , Failover count: 0
node0  0        lost                 n/a     n/a      n/a
node1  1        disabled             no      no       None\

Redundancy group: 1 , Failover count: 1
node0  100      primary              no      no       None
node1  0        hold

4- Disconnect link from node0 to cr1-codfw all traffic switched to node1 using interface xe-7/2/0 which it connected to cr2-codfw

I am leaving the task open since I have to update all the cable ID's.

I added the second fabric link xe-0/2/2

Fabric interfaces: 
    Name    Child-interface    Status                    Security
                               (Physical/Monitored)
    fab0    xe-0/2/2           Up   / Up                 Disabled   
    fab0    xe-0/2/3           Up   / Up                 Disabled   
    fab1    xe-7/2/2           Up   / Up                 Disabled   
    fab1    xe-7/2/3           Up   / Up                 Disabled

In T401297#11203591, @Papaul wrote:

2- Disconnect first and second HA links, lost connectivity to node 0 (unknown state) and node 1 pass to ineligible state after some minutes about 1 minute node 1 went into disabled state but we still have connectivity from eqiad to codfw

I'm not sure that's enough to conclude the HA links aren't needed for anything. I'd have expected to see logs of the state taken on both PFWs during the time, the routing on the CRs showing routes recieved etc, and the L2 forwarding table on the attached switches showing where virtual MACs were learnt. We also need to compare that to the scneario in which we just power off one of the units.

Can we set up another date to redo this test in a more structured way and record the results in more detail? If we decide they aren't needed - and then there is some future hit to fundraising traffic if something fails - then it will be on our heads. I'm not really comfortable to take that responsibility so far.