Page MenuHomePhabricator
Create Task
Maniphest T394021

Stage and configure new Juniper switches in codfw rows E/F
Closed, ResolvedPublic
Actions

Description

The following new devices have been racked and connected to management in codfw:

ssw1-e1-codfw
ssw1-f1-codfw
lsw1-e1-codfw
lsw1-e3-codfw
lsw1-f1-codfw
lsw1-f3-codfw

The next step is to get them configured and working as we need. These devices will be configured in a spine/leaf arrangement similar to rows A/B and C/D in codfw, however they will not use VXLAN/EVPN. Instead we will do dual-stack on all links, with OSPF & OSFP3 across them, and unicast IBGP between loopbacks in each address family. The plan is to configure some of this manually to nail down the correct configuration, then work backwards from there to adjust out automation to support both routing designs

Connectivity to the core routers on site requires additional ports that will become available when T393552 is complete.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedcmooneyT394021 Stage and configure new Juniper switches in codfw rows E/F
 ResolvedcmooneyT393552 codfw: setup MPC10E-10C and SCBE3
Unknown Object (Task)
Unknown Object (Task)

Event Timeline

cmooney created this task.May 13 2025, 12:13 PM
cmooney triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 13 2025, 12:13 PM
cmooney added a subtask: T393552: codfw: setup MPC10E-10C and SCBE3.May 13 2025, 12:13 PM
gerritbot added a comment.May 13 2025, 1:17 PM

Change #1145194 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Network: add puppet data for new devices and networks codfw expansion

https://gerrit.wikimedia.org/r/1145194

gerritbot added a project: Patch-For-Review.May 13 2025, 1:17 PM
gerritbot added a comment.May 13 2025, 3:15 PM

Change #1145246 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Add new INCLUDE statements in 0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa zone

https://gerrit.wikimedia.org/r/1145246

gerritbot added a comment.May 13 2025, 3:32 PM

Change #1145246 merged by Cathal Mooney:

[operations/dns@master] Add new INCLUDE statements in 0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa zone

https://gerrit.wikimedia.org/r/1145246

cmooney added a parent task: Unknown Object (Task).May 14 2025, 10:07 AM
cmooney added a subtask: Unknown Object (Task).
gerritbot added a comment.May 14 2025, 2:21 PM

Change #1145977 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Enable link-protection and BFD on OSPF links on EVPN switches

https://gerrit.wikimedia.org/r/1145977

gerritbot added a comment.May 15 2025, 10:05 AM

Change #1145977 merged by jenkins-bot:

[operations/homer/public@master] Enable link-protection on OSPF links on EVPN switches

https://gerrit.wikimedia.org/r/1145977

gerritbot added a comment.May 15 2025, 3:50 PM

Change #1146662 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Add EBGP between codfw row A-D spines and row E/F spines

https://gerrit.wikimedia.org/r/1146662

Stashbot added a comment.May 15 2025, 4:35 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-15T16:35:30Z] <topranks> add bgp peerings from codfw row A-D switches to new spines in rows E/F T394021

gerritbot added a comment.May 15 2025, 5:21 PM

Change #1146662 merged by jenkins-bot:

[operations/homer/public@master] Add EBGP between codfw row A-D spines and row E/F spines

https://gerrit.wikimedia.org/r/1146662

Stashbot added a comment.May 15 2025, 5:23 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-15T17:23:16Z] <topranks> add remaining bgp peerings from codfw row A-D switches to new spines in rows E/F T394021

gerritbot added a comment.May 16 2025, 2:49 PM

Change #1147014 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] New device additions for codfw expansion plus policy changes

https://gerrit.wikimedia.org/r/1147014

cmooney closed subtask T393552: codfw: setup MPC10E-10C and SCBE3 as Resolved.May 20 2025, 6:24 PM
cmooney reopened subtask T393552: codfw: setup MPC10E-10C and SCBE3 as Open.May 20 2025, 6:54 PM
cmooney closed subtask T393552: codfw: setup MPC10E-10C and SCBE3 as Resolved.May 21 2025, 9:38 AM
gerritbot added a comment.May 21 2025, 11:37 AM

Change #1148842 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Add new INCLUDE statement for 2620:0:860:139::/64 reverse

https://gerrit.wikimedia.org/r/1148842

gerritbot added a comment.May 21 2025, 11:55 AM

Change #1148842 merged by Cathal Mooney:

[operations/dns@master] Add new INCLUDE statement for 2620:0:860:139::/64 reverse

https://gerrit.wikimedia.org/r/1148842

gerritbot added a comment.May 21 2025, 1:10 PM

Change #1147014 merged by jenkins-bot:

[operations/homer/public@master] New device additions for codfw expansion plus policy changes

https://gerrit.wikimedia.org/r/1147014

Stashbot added a comment.May 21 2025, 1:16 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-21T13:16:49Z] <cmooney@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "import new switches from netbox to hiera now they are status active - cmooney@cumin1003 - T394021"

Stashbot added a comment.May 21 2025, 1:17 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-21T13:17:44Z] <cmooney@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "import new switches from netbox to hiera now they are status active - cmooney@cumin1003 - T394021"

gerritbot added a comment.May 21 2025, 1:22 PM

Change #1148867 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Codfw Spine EBGP: Fix typo in peer IP on ssw1-e1-codfw

https://gerrit.wikimedia.org/r/1148867

gerritbot added a comment.May 21 2025, 1:25 PM

Change #1148867 merged by jenkins-bot:

[operations/homer/public@master] Codfw Spine EBGP: Fix typo in peer IP on ssw1-e1-codfw

https://gerrit.wikimedia.org/r/1148867

gerritbot added a comment.May 21 2025, 4:33 PM

Change #1145194 merged by Cathal Mooney:

[operations/puppet@production] Network: add puppet data for new devices and networks codfw expansion

https://gerrit.wikimedia.org/r/1145194

Maintenance_bot removed a project: Patch-For-Review.May 21 2025, 5:30 PM
gerritbot added a comment.May 26 2025, 12:03 PM

Change #1150642 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Add entry for cagefive2* hosts in site.pp

https://gerrit.wikimedia.org/r/1150642

gerritbot added a project: Patch-For-Review.May 26 2025, 12:03 PM
cmooney added a subscriber: Jhancock.wm.May 26 2025, 12:22 PM

@Jhancock.wm hey I'm having some problems reaching cagefive2001 over management. The IP it is assigned is not responding to ping when I try? I know this had an old mgmt IP (10.193.1.94, re-used since for wikikube-worker2251), but doing a few tests it does not appear to be listening on this IP either right now.

Would you be able to set the server iDRAC IP to 10.193.3.224/16 manually and I can re-try?

gerritbot added a comment.May 27 2025, 8:26 AM

Change #1150642 abandoned by Cathal Mooney:

[operations/puppet@production] Add entry for cagefive2* hosts in site.pp

Reason:

will try to rename the hosts

https://gerrit.wikimedia.org/r/1150642

Maintenance_bot removed a project: Patch-For-Review.May 27 2025, 8:31 AM
Jhancock.wm added a comment.May 27 2025, 3:49 PM

@cmooney got it set and confirmed it pings

cmooney added a comment.May 27 2025, 3:50 PM
In T394021#10859945, @Jhancock.wm wrote:

@cmooney got it set and confirmed it pings

awesome, thank you!

ops-monitoring-bot added a comment.May 29 2025, 3:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host sretest2007.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.May 29 2025, 4:17 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host sretest2007.codfw.wmnet with OS bookworm completed:

  • sretest2007 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505291556_cmooney_780917_sretest2007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
gerritbot added a comment.May 30 2025, 4:59 PM

Change #1152301 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Add BGP session from cr1-codfw to ssw1-e1-codfw and remove nokia

https://gerrit.wikimedia.org/r/1152301

gerritbot added a project: Patch-For-Review.May 30 2025, 4:59 PM
gerritbot added a comment.May 30 2025, 5:02 PM

Change #1152301 merged by jenkins-bot:

[operations/homer/public@master] Add BGP session from cr1-codfw to ssw1-e1-codfw and remove nokia

https://gerrit.wikimedia.org/r/1152301

Maintenance_bot removed a project: Patch-For-Review.May 30 2025, 5:31 PM
gerritbot added a comment.Jun 3 2025, 11:14 AM

Change #1153120 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Codfw expansion: enable BGP from cr2-codfw to ssw1-f1-codfw

https://gerrit.wikimedia.org/r/1153120

gerritbot added a project: Patch-For-Review.Jun 3 2025, 11:14 AM
gerritbot added a comment.Jun 3 2025, 12:33 PM

Change #1153120 merged by jenkins-bot:

[operations/homer/public@master] Codfw expansion: enable BGP from cr2-codfw to ssw1-f1-codfw

https://gerrit.wikimedia.org/r/1153120

gerritbot added a comment.Jun 3 2025, 12:45 PM

Change #1153141 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Typo in bgp peer definition on cr2-codfw

https://gerrit.wikimedia.org/r/1153141

gerritbot added a comment.Jun 3 2025, 12:46 PM

Change #1153141 merged by jenkins-bot:

[operations/homer/public@master] Typo in bgp peer definition on cr2-codfw

https://gerrit.wikimedia.org/r/1153141

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2025, 1:30 PM
cmooney closed this task as Resolved.Jun 12 2025, 1:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits