@Jclark-ctr I think existing stock was used for the 100G links between the switches in codfw. I know we should have some spare 100GBase-CWDM4 (green handle) optics in eqiad but I think probably we only have one or two for spares?

If we don't have extra stock we can use without eating into our spares I'd recommend we order three of these for the job (incl. 1 spare):

https://www.fs.com/products/104216.html?attribute=10068&id=3763076

We should also use SFP28 modules for the 'fabric-link' between the two pfw's if possible, two of these (one for spare) ought to do the job:

https://www.fs.com/products/65860.html?attribute=9351&id=3838238

@RobH so looking at the options after discussion I think we need to do an fs.com (or alternative but with same options) order for the following:

Total cost is approx US$300 on fs.com so hopefully won't be a big issue, I think in the original budget there was some consideration for this looking at what Arzhel originally put down. The benefits of this are:

• We use 25G links from firewall to switches, 2.5 times more bandwidth
• We use two 10G links for the firewall to firewall fabric link, doubling the bandwidth and making it redundant
• We use 0.5m cables for the links between devices that are right on top of each other in the rack (much neater)

Nicely written plan !!

Fmsw connects directly to firewalls

We need to do the same in codfw before or after the eqiad migration so the two infras are as similar as possible (cc @Papaul

NOTE: The exact ports on the new switches for the last 4 links needs to be confirmed after we have a full-port plan for what servers will connect to what switch ports after the migration. Due to the QFX5120 requirement to configure port speeds in blocks of 4 we need to determine how many 1G and 10G server connections we will have before we can plan those links.

Same as above, it would be best if possible to assign the same "infra" ports as what we did in codfw.

In T377381#10240797, @ayounsi wrote:

Fmsw connects directly to firewalls

We need to do the same in codfw before or after the eqiad migration so the two infras are as similar as possible (cc @Papaul

Agreed yeah we should mirror in codfw, overall I think it's a benefit so we can test in eqiad and retro-fit in codfw if we're happy.

Same as above, it would be best if possible to assign the same "infra" ports as what we did in codfw.

The problem there is we may have a different mix of servers at 10G/1G on the switches, and different numbers of port-blocks. But yes provided there is no barrier I'll keep them the same.

In T377381#10242011, @cmooney wrote:

In T377381#10240797, @ayounsi wrote:

Fmsw connects directly to firewalls

We need to do the same in codfw before or after the eqiad migration so the two infras are as similar as possible (cc @Papaul

Agreed yeah we should mirror in codfw, overall I think it's a benefit so we can test in eqiad and retro-fit in codfw if we're happy.

Same as above, it would be best if possible to assign the same "infra" ports as what we did in codfw.

The problem there is we may have a different mix of servers at 10G/1G on the switches, and different numbers of port-blocks. But yes provided there is no barrier I'll keep them the same.

@cmooney thanks for putting this together. if you login to fasw2-c8a/b-codfw you will see that i have all the 1G using the first 12 ports on the switches and the 10G all the ports starting from 13. I had to moving servers around to get to this result.
TIP: first i count the number of servers with only 1G NIC's those will be the servers using 1G in my case 12 servers to i reserved the frist ports for those
second i start planing on relocating those that were not within 0-12U once i had the plan in place i ping Jeff and started relocating those servers even before the migration process and keeping the same port on the old switch but on the new switch i set them up to use the rack U space.

In T377381#10242050, @Papaul wrote:

TIP: first i count the number of servers with only 1G NIC's those will be the servers using 1G in my case 12 servers to i reserved the frist ports for those
second i start planing on relocating those that were not within 0-12U once i had the plan in place i ping Jeff and started relocating those servers even before the migration process and keeping the same port on the old switch but on the new switch i set them up to use the rack U space.

Cool thanks @Papaul we'll try and follow the same pattern in eqiad that makes sense. I'll ask John and Valerie if they want to move the servers to match the U to port number leave the decision to them, but we'll configure the first N blocks on the switch as 1G to match.

I spoke to @Jgreen earlier about the setup and I think the above plan should be workable, provided we carry out the works during fundraising maintenance week starting Mon Nov 4th.

@Jclark-ctr, @VRiley-WMF if we could get the two new firewalls installed next week that would be great. Then I'd propose the following schedule on the week of Nov 4th:

Open to suggestions but I think that makes sense if it works for everyone else.

@cmooney Dallas did a survey of existing servers at eqiad, and none of them have 10G interfaces. So 10G port needs will be just for new hardware.

In T377381#10246582, @Jgreen wrote:

@cmooney Dallas did a survey of existing servers at eqiad, and none of them have 10G interfaces. So 10G port needs will be just for new hardware.

Huh ok. That's a bit of a shame in some ways but no problems. It does make things a little easier as I guess we can leave all server ports the same on the new switches (i.e. if a server is connected to ge-0/0/7 on the old switches we will connect it to the same port on the new)? We can move things around of course if there is any benefit in that.

I'll provisionally also allocate ports 40-43 for 10G connections (the 2x10G trunk from new fasw2-c1a-eqiad to existing fasw-c-eqiad during migration), and ports 44-47 at 25G (the uplinks from new switches to new firewalls).

That's a bit of a shame in some ways but no problems

We'll get there next procurement cycle! Note that we do have a few new servers that will be 10G, but I don't think they're racked yet.

In T377381#10246886, @Jgreen wrote:

That's a bit of a shame in some ways but no problems

We'll get there next procurement cycle! Note that we do have a few new servers that will be 10G, but I don't think they're racked yet.

Are there tasks for those or a list anywhere? Are they additional servers or replacing existing ones? Given how tight we are on free ports it might be worth to plan out where we'll connect them (and what blocks of 4 should be moved to 10G) in advance of this move. Thanks.

There are 6 servers being replaced:
T369565: Q1:rack/setup/install frnetmon1002, pay-lb1001, pay-lb1002
T369947: Q1:rack/setup/install frban1002
T369947: Q1:rack/setup/install frban1002

Plus 3 new servers:
T367820: Q1:rack/setup/install franio100[1-3]

@cmooney Step 1: Firewall Installation & Cabling is complete

Since we have racked the new switches already would it be ok if we start cabling for step 3?

In T377381#10252289, @Jclark-ctr wrote:

@cmooney Step 1: Firewall Installation & Cabling is complete

Since we have racked the new switches already would it be ok if we start cabling for step 3?

Thanks @Jclark-ctr, you moved msw-c1-eqiad to make room for them? That's great nice thinking.

Yes we can move on to do the cabling for the switches as well, did the fs.com order (T377481) arrive? I'll update the plan later in the day based on not having to stagger it so much.

We can also likely do all the work now during a single window, as I'll be able to pre-configure and validate all the new gear and links in advance. What day works for that? Tues Nov 5th is a candidate but I know that's election day in the US, if Wed 6th suits better we can do it then. CC @Jgreen

@cmooney all cables have been connected for Step 2: Initial cabling for the new devices for switches and firewalls. Ping me if anything else comes up or is needed

In T377381#10256437, @Jclark-ctr wrote:

@cmooney all cables have been connected for Step 2: Initial cabling for the new devices for switches and firewalls. Ping me if anything else comes up or is needed

Great thanks John, I'll check all the links and let you know if all looks good. Like you said let's schedule the works for Wed Nov 6th if we can, @Jgreen does that work for you?

@cmooney @Jclark-ctr There has been a request to push our maintenance week back one week if possible. Would you all be ok with doing the work on the following week (Nov 11-15)?

Fr-Tech is ok with the shift but know you all would be impacted. Let us know if this would work of if you need to keep the work on the week of Nov 4-8.

In T377381#10263676, @Dwisehaupt wrote:

@cmooney @Jclark-ctr There has been a request to push our maintenance week back one week if possible. Would you all be ok with doing the work on the following week (Nov 11-15)?

@Dwisehaupt absolutely that is no problem. Actually suits me much better as I'm at a conference this week so I can use next week to get things prepped instead.

Any particular day to do this? Monday is normally fairly busy for me with meetings, so perhaps Tuesday Nov 12th? @Jclark-ctr are you around that day?

@cmooney Sorry for delay i am good for nov 12th

100GBase-CWDM4 (green handle) we have a few extra from new cage buildout did you have a number of what we need? I am pretty sure we have atleast x5

@cmooney @Jclark-ctr Wonderful, thanks for the flexibility. We should get confirmation of the shift in date from the FR folks tomorrow afternoon PDT. We will let you know when it is confirmed.

@cmooney @Jclark-ctr Got confirmation that the date shift is good. We are all set to do the network update on Nov 12. Thanks.

@cmooney fyi i have 10x of the 100g green handled optics

In T377381#10274577, @Jclark-ctr wrote:

@cmooney fyi i have 10x of the 100g green handled optics

@Jclark-ctr sorry for the delayed response. I think we are good for those, they were a backup plan instead of the short DACs but I think we are ok. Good to know we have some spares anyway.

In terms of the wiring I've gone through everything and we're mostly ok, just one problem and one thing to clarify.

Firewall Control Plane Ports

These two are reported as down (no SFP/cable inserted) on both firewall devices:

root> show chassis cluster control-interface status 
Warning: No SFP present on control ports!.

Serial Console Port Connections

I was able to access all 4 devices over serial console, on the ports I listed. I wanted to check though, the port numbers I gave were based on what looked free in Netbox, however the two below already had config - other device names - configured on the OpenGear when I actually logged on to it:

These were the names on the ports:

When I check in Netbox neither of those devices have anything connected to their console port either (here and here). So I wanted to double check the situation, did you remove any cables from the SCS to connect the new gear? Either way we'll need to get the two devices above connected to a console server so I can create a task for that.

Other than that all looking good. I can't start any work on the firewalls without the control ports connected though so if you could have a look at that asap it would be appreciated. We can also try copper SFPs and regular RJ45 patches if the 1G DACs we bought aren't working.

@Jclark-ctr could you also let me know what ports on the fmsw these two were plugged into?

Doesn't need to go in netbox but I'll make a note on the port description. Thanks.

In T377381#10250655, @Jgreen wrote:

There are 6 servers being replaced:
T369565: Q1:rack/setup/install frnetmon1002, pay-lb1001, pay-lb1002
T369947: Q1:rack/setup/install frban1002
T369947: Q1:rack/setup/install frban1002

Plus 3 new servers:
T367820: Q1:rack/setup/install franio100[1-3]

Thanks @Jgreen . Looking at the existing ports on the switch I think it might make sense if we change the port assignment of these two hosts when we do the migration next week:

My thinking here is that the block of 4 ports from 36-39 only has a single used port right now, for civi1002. If we instead connect port 36 to something that is going to be replaced in the near future it means we can switch that entire block of 4 from 1G to 10G when frban1001 is decom'd. Which should give us more flexibility in general to add the new hosts and deal with the port block restriction.

Let me know if that sounds ok, I can pre-configure the new switches for that if so. Cheers.

All, just to be aware I hit another snag this evening which may be problematic.

When trying to configure the new trunk ports that go from the existing switch stack in rack C1 (fasw-c-eqiad) to newly installed fasw2-c1b-eqiad I am getting an error whereby the virtual-chassis cannot apply the configuration to the second switch:

cmooney@fasw-c-eqiad# show | compare 
[edit chassis]
+   aggregated-devices {
+       ethernet {
+           device-count 1;
+       }
+   }
[edit interfaces]
+   xe-0/2/1 {
+       description "fasw-c1a-eqiad xe-0/0/42";
+       ether-options {
+           802.3ad ae0;
+       }
+   }
+   xe-1/2/1 {
+       description "fasw-c1a-eqiad xe-0/0/43";
+       ether-options {
+           802.3ad ae0;
+       }
+   }
+   ae0 {
+       description "2x10G Trunk to fasw-c1a-eqiad";
+       mtu 9192;
+       aggregated-ether-options {
+           lacp {
+               active;
+               periodic fast;
+           }
+       }
+       unit 0 {
+           family ethernet-switching {
+               interface-mode trunk;
+               vlan {
+                   members [ frack-payments1-c-eqiad frack-administration1-c-eqiad frack-fundraising1-c-eqiad frack-bastion1-c-eqiad frack-listenerdmz1-c-eqiad ];
+               }
+           }
+       }
+   }

{master:0}[edit]
cmooney@fasw-c-eqiad# commit check 
fpc0: 
configuration check succeeds
fpc1: 
error: Could not open config database: Could not open database schema
Failed opening DB for traceoptions read ... Exiting
  'Could not open database schema'
    db-open failed
error: configuration check-out failed
fpc0: 
error: remote commit-configuration failed on fpc1
error: configuration check-out failed

This is kind of worrying. I will open a TAC case with Juniper on the matter in the morning and see what they say, however I fear the only potential remedies may be intrusive. I'd guess if we reboot both of the old switches it would fix it, so we can try this on Tuesday, however if it still fails at that point we could be in trouble.

Options

If we can't modify the existing switch config to connect to the new ones then we won't be able to do the gradual move of all servers from old to new switches as planned.

It *may* be possible for us to connect all 4 switches to the new firewall stack at once, members of reth0, but I will need to look into that in more detail to see exactly how it would operate.

We could also decide to try and migrate all the servers "big bang" style from the old to new switches during the move next week. I don't think that would be impossible (port numbers are staying the same except the one I mentioned in last comment), however it definitely would mean more disruption than we planned for, and more checks afterwards (i.e. both server links are working).

Thanks @Jgreen . Looking at the existing ports on the switch I think it might make sense if we change the port assignment of these two hosts when we do the migration next week:

Host	Current switch port	New switch port
frban1001	0	36
civi1002	36	0

My thinking here is that the block of 4 ports from 36-39 only has a single used port right now, for civi1002. If we instead connect port 36 to something that is going to be replaced in the near future it means we can switch that entire block of 4 from 1G to 10G when frban1001 is decom'd. Which should give us more flexibility in general to add the new hosts and deal with the port block restriction.

Let me know if that sounds ok, I can pre-configure the new switches for that if so. Cheers.

Thanks @cmooney. This should be fine. We don't have anything that we track on a switch port basis.

@cmooney replaced 1g dac cables with sfpt and cat6 cables.

These two switches have been removed from racks.
asw2-d5-eqiad
cloudsw2-c8-eqiad

In T377381#10297595, @Jclark-ctr wrote:

These two switches have been removed from racks.
asw2-d5-eqiad
cloudsw2-c8-eqiad

Ah ok my bad on that, yep you are right.

@cmooney replaced 1g dac cables with sfpt and cat6 cables.

As discussed on irc this immediately made no difference, but it seems the actual problem was the system doesn't display the status of those "cluster" ports until it's put into cluster mode. The error message saying "no sfp inserted" threw me, not at all easy to understand. Looks ok now thanks!

@Jclark-ctr only other thing right now I need from you is to add the cable labels in Netbox. I've added all the links to it now. No rush at all, Netbox needs a label so I just used dummy ones.

https://netbox.wikimedia.org/dcim/cables/?q=changeme_frack&site_id=6

@Jclark-ctr as discussed I believe we should have a load of copper SFPs from T369557.

We need one of these inserted into ports 0 through 36 on each of the new switches (fasw2-c1a and fasw2-c1b). Looks like there are some in there already but not all those ports are covered.

Thanks.

Change #1088373 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Add puppet entries for new fundraising switches in eqiad

https://gerrit.wikimedia.org/r/1088373

Change #1088535 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Move idle-timeout under login to the dedicated login template

https://gerrit.wikimedia.org/r/1088535

Change #1088373 merged by Cathal Mooney:

[operations/puppet@production] Add puppet entries for new fundraising switches in eqiad

https://gerrit.wikimedia.org/r/1088373

Change #1088537 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Remove pfw3-eqiad and replace with pfw1-eqiad

https://gerrit.wikimedia.org/r/1088537

@Jgreen @Dwisehaupt I was doing some prep work on T377996 - looking at step 1 to import the existing data into Netbox (step 2 will be to get the new server provisioning flow in place which I'll tackle next).

I noticed some devices which I couldn't find a MAC address for, nor log on etc. I think these may be decom'd machines, none of them are in Netbox either.

Both ports down on switch:

ge-0/0/13       up    down payments2 eth1
ge-1/0/13       up    down payments2

ge-0/0/14       up    down payments4 eth1
ge-1/0/14       up    down payments4

At least one port is up on switch, and there is a DNS entry, but no MAC learnt on switch:

ge-0/0/12       up    up   frdb1002 - no-mon
ge-1/0/12       up    down frdb1002 eth1

At least one port up on switch, but no DNS entry and no MAC learnt:

ge-0/0/5        up    up   frdev1001
ge-1/0/5        up    up   frdev1001 eth1

ge-0/0/7        up    up   frpm1001
ge-1/0/7        up    up   frpm1001 eth1

ge-0/0/8        up    up   frlog1001
ge-1/0/8        up    down frlog1001 eth1

ge-0/0/17       up    up   frav1002:eth0
ge-1/0/17       up    up   frav1002:eth1

ge-0/0/20       up    up   civi1001 eth1
ge-1/0/20       up    up   civi1001

ge-0/0/22       up    up   frmon1001
ge-1/0/22       up    up   frmon1001

ge-0/0/23       up    up   frbast1001
ge-1/0/23       up    up   frbast1001

ge-0/0/25       up    up   frpig1001
ge-1/0/25       up    up   frpig1001

Can we confirm the status of these? Similar to the discussions above about port block speeds if these aren't needed we should probably not configure ports on the new switch for them, and shuffle the remaining 1G servers into the first N ports removing the gaps. This will leave us the maximum number of contiguous free ports to configure into blocks of 4x10G for the new servers going in and future migrations.

I'd a chat with @Jgreen on irc about the above and he confirmed all those hosts are decommed. We're a little perplexed as to what the switch ports are plugged into that makes them show "up", as the hosts don't appear to be in the rack, but for now we will plan to not migrate those ports to the new switches.

@Jclark-ctr that means we only need copper SFPs in the ports 0-27 on the new switches. But when we migrate hosts we do not need to move all cables, and unfortunately for some of the hosts we will move to a different port from old switch (same pattern on both switches).

I've adjusted Netbox to match the above and have pre-configured the new switches for the updated allocations.

This will save a lot of hassle when connecting the new hosts we have as we should have enough free 10Gs for them without shuffling things around.

@cmooney thanks for the list I have populated both new switches up to port 27 with sfp-t

Change #1088535 merged by jenkins-bot:

[operations/homer/public@master] Move idle-timeout under login to the dedicated login template

https://gerrit.wikimedia.org/r/1088535

@Jgreen @Dwisehaupt I think we have broadly two options for how to proceed today:

Option 1: Begin with a reboot of fasw-c-eqiad to try to resolve disk/config issue

Doing this will take all the hosts down for the time the switches take to reboot, approx 15 mins. Once it comes back up if we are lucky we will be able to configure/enable the trunk from new switch fasw2-c1a-eqiad to it, which will allow us to migrate server links 1-by-1 from the old switch to new switch at our own pace.

We then proceed and migrate the live traffic path to use the new firewalls, and make the inside connection from those to the new switches the active path. Servers can stay where they are as there is a traffic path from old switch -> new switch -> new firewalls.

Option 2: Move the first half of the server links to fasw2-c1b-eqiad as step 1

This is basically the option Jeff suggested on irc. We can safely move all the server uplinks that currently land on fasw-c1b-eqiad to fasw2-c1b-eqiad in advance of any other work. We would leave the switch ports in an admin DOWN state on the new switch, so none of the moved links would be live immediately after the move. Any servers that had been using their second link as the active one will switch to using the primary (so all traffic flowing via old switch fasw-c1a-eqiad).

When this is done we can proceed to do the move of the core router uplinks from the old firewall cluster to the new one, which as expected will disrupt all traffic. As part of that work what we will do is:

• Power down the two old switches, fasw-c1a-eqiad and fasw-c1b-eqiad
• Enable the switch ports on fasw2-c1b-eqiad connected to the server links we just moved.
• Update the IP interface configuration on the new firewall cluster (pfw1-eqiad) to make it gateway for all the frack vlans

At which point the server secondary ports should be active, with traffic flowing via the new fasw2-c1b switch, and externally through the new firewall cluster. We can then move the remaining server uplinks from (now offline) fasw-c1a-eqiad to fasw2-c1a-eqiad and the full migration is done.

Icinga downtime and Alertmanager silence (ID=fd1b13c3-25ae-42de-a138-bb1a3989c0b4) set by cmooney@cumin1002 for 4:00:00 on 1 host(s) and their services with reason: fundraising tech migration to new equipment

fasw-c-eqiad

Icinga downtime and Alertmanager silence (ID=6d3e8237-b81b-47ec-a63c-afd9f7859ae7) set by cmooney@cumin1002 for 4:00:00 on 3 host(s) and their services with reason: fundraising tech migration to new equipment

cr[1-2]-eqiad,pfw3-eqiad

Mentioned in SAL (#wikimedia-operations) [2024-11-12T15:16:23Z] <topranks> moving fundraising links in eqiad from old to new firewall cluster and switches (T377381)

Change #1088537 merged by Cathal Mooney:

[operations/puppet@production] Remove pfw3-eqiad and replace with pfw1-eqiad

https://gerrit.wikimedia.org/r/1088537

Migration work is now complete, bastion and all hosts are reachable again following the moves. BGP is established to both core routers from the new firewall pair and everything looks good. Management is reachable directly from the firewalls on reth1, IPsec tunnel to codfw is working with the config built by automation also.

Thanks all for their help on this!

Change #1091182 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Replace fasw-c-eqiad with new fasw2

https://gerrit.wikimedia.org/r/1091182

Change #1091228 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Remove old fr-tech switch stack from rancid backups

https://gerrit.wikimedia.org/r/1091228

Change #1091228 abandoned by Cathal Mooney:

[operations/puppet@production] Remove old fr-tech switch stack from rancid backups

Reason:

forgot to remove device from common.yaml too

https://gerrit.wikimedia.org/r/1091228

Change #1091182 merged by Cathal Mooney:

[operations/puppet@production] Remove old fasw-c-eqiad from monitoring

https://gerrit.wikimedia.org/r/1091182