Page MenuHomePhabricator

Q2:rack/setup E8/F8 new leaf switches
Closed, ResolvedPublic

Description

This task will track the racking, setup, and OS installation of E8/F8 new leaf switches ordered via T380017

Hostname / Racking / Installation Details

netops will need to detail how to cable these into the spine/leaf configuration.

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

switch1
  • Receive in system on procurement task T380017 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Configure switch with Homer and add to monitoring
switch2
  • Receive in system on procurement task T380017 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Configure switch with Homer and add to monitoring

Event Timeline

@ayounsi or @cmooney: These two switches will arrive in December. Would one of you be able tot update this task with the cabling directions to join these into the existing spine/leaf setup in rows E/F?

RobH added a parent task: Unknown Object (Task).

Sure, as usual for power/console/mgmt.
Regarding production ports :
On the ssw1 side: use et-0/0/7` towards e8 and et-0/0/15 towards f8. They're currently used by the links to the dell switches, so no need to re-run new cables.

On the lsw side : use et-0/0/54 to ssw1-e1 and et-0/0/55 to ssw1-f1

All using QSFP-100G-CWDM4 optics.

@Jclark-ctr @VRiley-WMF the 2 switches are received in coupa but are missing in netbox. if there are not ready to be racked yet, can you please enter them in netbox with asset tags as planned so I can update the accounting sheet?

Thanks

@VRiley-WMF I have not seen these in the data center yet but you updated ticket Jan 10 2025 almost 2 months ago? Receiving ticket is still open also but they have been received in coupa?

Thanks guys. Please ping me when these are in Netbox and I will add the links, IPs, vlans etc. and begin the process of commissioning them.

Also note it is confusing we have two "device_types" in Netbox for these (corresponding to an older and newer Juniper product code for them). I assume these will be "QFX5120-48Y-AFI2" (note the '2' at the end), so please use that device type.

Lastly please call these new switches lsw1-e8-eqiad and lsw1-f8-eqiad in Netbox. We'll need to either have deleted the Dell switches from Netbox before we will be able to do that, or rename the Dell's to something else.

Will be adding these into netbox shortly

Change #1125488 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Add new Juniper leaf switches eqiad E8/F8 to IBGP cluster

https://gerrit.wikimedia.org/r/1125488

Will be adding these into netbox shortly

Cool I can see them there. FWIW I added the spine links now in Netbox and added the IPs to them. Next week I'll have a go at getting them configured and ready for servers.

Change #1125488 abandoned by Cathal Mooney:

[operations/homer/public@master] Add new Juniper leaf switches eqiad E8/F8 to IBGP cluster

Reason:

messed this up somehow

https://gerrit.wikimedia.org/r/1125488

Change #1126136 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Add new switches eqiad racks E8/F8

https://gerrit.wikimedia.org/r/1126136

Change #1126136 merged by jenkins-bot:

[operations/homer/public@master] Add new switches eqiad racks E8/F8

https://gerrit.wikimedia.org/r/1126136

@VRiley-WMF I've fixed up the scs links in netbox now, moving the cables to the new devices.

To set the boxes to 'active' (needed before I could run Homer) I had to set the location for them, but tbh I didn't know what it was. Can you go in and make sure they are set to face the correct direction and at the right U height? And then run the sre.puppet.sync-netbox-hiera cookbook to sync the proper location to puppet? Thanks.

Mentioned in SAL (#wikimedia-operations) [2025-03-11T11:23:48Z] <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "add new network switches - cmooney@cumin1002 - T382017"

Mentioned in SAL (#wikimedia-operations) [2025-03-11T11:24:13Z] <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "add new network switches - cmooney@cumin1002 - T382017"

@RobH did Myriad send us the license files for these two new QFX5120-48Y switches?

Mentioned in SAL (#wikimedia-operations) [2025-03-11T11:31:17Z] <topranks> enable connections from ssw1-e1 and ssw1-f1 to new top-of-rack switches lsw1-e8 and lsw1-f8 in eqiad T382017

Change #1126534 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Add base monitoring elements for new top-of-rack switches eqiad E8/F8

https://gerrit.wikimedia.org/r/1126534

Change #1126534 merged by Cathal Mooney:

[operations/puppet@production] Add base monitoring elements for new top-of-rack switches eqiad E8/F8

https://gerrit.wikimedia.org/r/1126534

Change #1127129 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Add overlay vrf loopback IPs to hieradata for new eqiad leaf switches

https://gerrit.wikimedia.org/r/1127129

Change #1127129 merged by Cathal Mooney:

[operations/puppet@production] Add overlay vrf loopback IPs to hieradata for new eqiad leaf switches

https://gerrit.wikimedia.org/r/1127129

Change #1127134 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] LVS: Add new sub-interfaces to LVS in eqiad for rack e8 and f8 vlans

https://gerrit.wikimedia.org/r/1127134

These have been racked and in use is anything else needed for you for ticket?

VRiley-WMF updated the task description. (Show Details)

Change #1144666 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Add Eqiad rack E8 and F8 vlan interfaces on eqiad lvs hosts

https://gerrit.wikimedia.org/r/1144666

Change #1144666 merged by Cathal Mooney:

[operations/puppet@production] Add Eqiad rack E8 and F8 vlan interfaces on eqiad lvs hosts

https://gerrit.wikimedia.org/r/1144666

Change #1145098 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] Revert^2 "Add Eqiad rack E8 and F8 vlan interfaces on eqiad lvs hosts"

https://gerrit.wikimedia.org/r/1145098

Change #1145099 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Add Eqiad rack E8 and F8 vlan interfaces on eqiad lvs hosts

https://gerrit.wikimedia.org/r/1145099

Change #1145098 abandoned by Vgutierrez:

[operations/puppet@production] Revert^2 "Add Eqiad rack E8 and F8 vlan interfaces on eqiad lvs hosts"

Reason:

work continuing on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1145099

https://gerrit.wikimedia.org/r/1145098

Change #1145099 merged by Vgutierrez:

[operations/puppet@production] lvs: add eqiad rack E8 and F8 vlan interfaces on eqiad lvs hosts

https://gerrit.wikimedia.org/r/1145099

Change #1127134 abandoned by Cathal Mooney:

[operations/puppet@production] LVS: Add new sub-interfaces to LVS in eqiad for rack e8 and f8 vlans

Reason:

was forgotten, then done in another emergency patch for low-traffic LVS's when there was an issue.

https://gerrit.wikimedia.org/r/1127134