Page MenuHomePhabricator
Create Task
Maniphest T396063

Eqiad: row C/D switch refresh
Open, MediumPublic
Actions

Description

High level placeholder task for the purchase/installation/configuration of new switches in eqiad rows C and D, as well as the migration of hosts to the new top-of-racks and decommissioning of the old hardware.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Include statements in reverse zones for new subnetsoperations/dnsmaster+211 -4
ssw1-d8-eqiad: add bgp peerings to CR and Juniper spinesoperations/homer/publicmaster+7 -0
Reverse include file statement for 2620:0:861:fe17::/64operations/dnsmaster+4 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT396063 Eqiad: row C/D switch refresh
Unknown Object (Task)
 ResolvedJclark-ctrT400780 eqiad row C/D upgrade racking task
 ResolvedJclark-ctrT396065 Eqiad: row C/D switch refresh cabling task
 ResolvedJclark-ctrT401238 Eqiad: row C/D switch refresh cabling task ssw1-d1-eqiad
 ResolvedVRiley-WMFT401240 Eqiad: row C/D switch refresh cabling task ssw1-d8-eqiad
 ResolvedcmooneyT402588 Eqiad: row C/D switch refresh configuration task
Unknown Object (Task)
 ResolvedVRiley-WMFT405499 Remove lvs1018 L2 link to ssw1-e1-eqiad
 ResolvedcmooneyT405558 Nokia: add new switches in eqiad/codfw to monitoring and make 'active'
 ResolvedJclark-ctrT405560 Eqiad C/D refresh: 2 x test hosts for config validation
 OpenJclark-ctrT404609 eqiad: rows C/D Upgrade Tracking
 ResolvedJclark-ctrT405021 eqiad row C/D DC Ops host migrations
 ResolvedJclark-ctrT405623 eqiad row C/D Traffic host migrations
 ResolvedJclark-ctrT405647 eqiad row C/D Machine Learning host migrations
 ResolvedJclark-ctrT405940 eqiad row C/D Collaboration Services host migrations
 ResolvedJclark-ctrT405942 eqiad row C/D Data Persistence host migrations
 ResolvedMarosteguiT410282 Switchover s1 master (db1184 -> db1163)
 ResolvedMarosteguiT410281 Switchover s7 master (db1181 -> db1236)
 ResolvedMarosteguiT410283 Switchover s3 master (db1189 -> db1223)
 ResolvedJclark-ctrT405943 eqiad row C/D Data Platform host migrations
 ResolvedRobHT405945 eqiad row C/D Infrastructure Foundations host migrations
 ResolvedRobHT405946 eqiad row C/D Observability host migrations
 ResolvedRobHT405948 eqiad row C/D Search Platform host migrations
 ResolvedJclark-ctrT405950 eqiad row C/D Service Ops host migrations
 ResolvedcmooneyT409800 Row C traffic outage Nov 11 2025
 ResolvedBTullisT409924 High pod latency affecting several dse-k8s-worker nodes in eqiad C/D rows
 ResolvedJclark-ctrT411025 eqiad row C/D cloud hosts pending migration
 ResolvedJclark-ctrT411781 lvs1018: remove cross-rack links to rows A, C and D
 ResolvedVRiley-WMFT412525 eqiad: rows C/D Upgrade Decom Asw Switches in Rows C & D
 ResolvedcmooneyT404146 Netbox: General updates for Nokia switch support
 ResolvedcmooneyT405637 Netbox: Update server provision script to support Nokia switches
 ResolvedcmooneyT405640 Netbox: Create script to allow multiple host migrations from old -> new switch
 ResolvedcmooneyT409286 Nokia L3 bugs [Oct 2025]
 ResolvedcmooneyT409178 Nokia SR-Linux ARP resolution bug on v24.10.x+
 ResolvedJclark-ctrT409288 Connect spare Nokia D2L switches to Spines in eqiad
 ResolvedJclark-ctrT411098 Move sretest1006 to rack D8 and connect to lswtest-d8-eqiad
 ResolvedJclark-ctrT412271 Decom Juniper EX/QFX switches in eqiad rows C/D
 ResolvedJclark-ctrT405579 Eqiad C/D refresh: move asw2-c-eqiad CR uplinks to Nokia switches
Unknown Object (Task)
 ResolvedcmooneyT409067 Eqiad C/D refresh: move asw2-d-eqiad CR uplinks to Nokia switches
 ResolvedJclark-ctrT405609 lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad
 ResolvedcmooneyT405628 lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad
 DeclinedJclark-ctrT410661 lvs1018: decom links to asw2-c2-eqiad and asw2-d7-eqiad

Event Timeline

cmooney created this task.Jun 4 2025, 5:54 PM
cmooney triaged this task as Medium priority.
cmooney added a subtask: Unknown Object (Task).
Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Jul 15 2025, 12:37 PM
Jclark-ctr subscribed.Jul 18 2025, 12:23 PM

Adjusted Mgmt in Rack D1 , D8 down to place spines in top of rack. Updated netbox and installed Rails

Jclark-ctr added a comment.Jul 24 2025, 12:57 PM

@cmooney @ ayounsi
This morning, I updated NetBox with names and locations for all refresh switches and ran two new console cables to SCS. I also verified cable IDs to ports—everything looks correct following the error found yesterday with racks D1 and D2. Assuming no mistakes were made with the cable labels during the original installation.

Switch cableid port
lsw1-c2-eqiad 2819 10
lsw1-c3-eqiad 2820 11
lsw1-c4-eqiad 2821 12
lsw1-c5-eqiad 2822 13
lsw1-c6-eqiad 2823 14
lsw1-c7-eqiad 2824 15
lsw1-d1-eqiad needs to be added to netbox id#1254 26
lsw1-d2-eqiad 3979 30
lsw1-d3-eqiad 3981 31
lsw1-d4-eqiad 3982 32
lsw1-d6-eqiad 3984 34
lsw1-d7-eqiad 3985 35
lsw1-d8-eqiad 3986 36
ssw1-d1-eqiad 3980 29
ssw1-d8-eqiad needs to be added to netboxid #1317 27

Jclark-ctr added a subtask: T400780: eqiad row C/D upgrade racking task.Jul 30 2025, 4:12 AM
cmooney added a subscriber: VRiley-WMF.Jul 30 2025, 10:20 AM
This comment was removed by cmooney.
cmooney added a subtask: T396065: Eqiad: row C/D switch refresh cabling task.Jul 30 2025, 12:06 PM
Jclark-ctr closed subtask T400780: eqiad row C/D upgrade racking task as Resolved.Jul 30 2025, 3:02 PM
cmooney added a subtask: Unknown Object (Task).Aug 20 2025, 7:32 PM
cmooney added a subtask: T402588: Eqiad: row C/D switch refresh configuration task.Aug 21 2025, 7:59 PM
cmooney added a subtask: Unknown Object (Task).Sep 9 2025, 4:06 PM
cmooney mentioned this in Unknown Object (Task).Sep 9 2025, 4:23 PM
gerritbot added a comment.Sep 24 2025, 1:41 PM

Change #1191025 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Reverse include file statement for 2620:0:861:fe17::/64

https://gerrit.wikimedia.org/r/1191025

gerritbot added a project: Patch-For-Review.Sep 24 2025, 1:41 PM
gerritbot added a comment.Sep 24 2025, 3:10 PM

Change #1191025 merged by Cathal Mooney:

[operations/dns@master] Reverse include file statement for 2620:0:861:fe17::/64

https://gerrit.wikimedia.org/r/1191025

Maintenance_bot removed a project: Patch-For-Review.Sep 24 2025, 3:32 PM
cmooney added a subtask: T405499: Remove lvs1018 L2 link to ssw1-e1-eqiad.Sep 24 2025, 5:21 PM
cmooney added a subtask: T405558: Nokia: add new switches in eqiad/codfw to monitoring and make 'active'.Sep 25 2025, 9:33 AM
cmooney added a subtask: T405560: Eqiad C/D refresh: 2 x test hosts for config validation.Sep 25 2025, 9:41 AM
cmooney added a subtask: T405562: Eqiad C/D refresh: move legacy switch uplinks to Nokias and migrate Vlan GWs.Sep 25 2025, 9:44 AM
gerritbot added a comment.Sep 25 2025, 2:20 PM

Change #1191429 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] ssw1-d8-eqiad: add bgp peerings to CR and Juniper spines

https://gerrit.wikimedia.org/r/1191429

gerritbot added a project: Patch-For-Review.Sep 25 2025, 2:20 PM
cmooney added a subtask: T405602: Eqiad row C/D switch refresh: LVS changes to support migration.Sep 25 2025, 3:00 PM
cmooney added a subtask: T404609: eqiad: rows C/D Upgrade Tracking.Sep 25 2025, 4:31 PM
cmooney removed a subtask: Unknown Object (Task).Sep 25 2025, 5:41 PM
cmooney added a subtask: T404146: Netbox: General updates for Nokia switch support.Sep 25 2025, 5:49 PM
Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Sep 30 2025, 12:10 PM
gerritbot added a comment.Oct 2 2025, 3:03 PM

Change #1191429 merged by jenkins-bot:

[operations/homer/public@master] ssw1-d8-eqiad: add bgp peerings to CR and Juniper spines

https://gerrit.wikimedia.org/r/1191429

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2025, 3:32 PM
RobH subscribed.Oct 8 2025, 3:56 PM
cmooney closed subtask T405499: Remove lvs1018 L2 link to ssw1-e1-eqiad as Resolved.Oct 16 2025, 5:24 PM
gerritbot added a comment.Oct 23 2025, 4:43 PM

Change #1198370 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Include statements in reverse zones for new subnets

https://gerrit.wikimedia.org/r/1198370

gerritbot added a project: Patch-For-Review.Oct 23 2025, 4:43 PM
gerritbot added a comment.Oct 24 2025, 10:10 AM

Change #1198370 merged by Cathal Mooney:

[operations/dns@master] Include statements in reverse zones for new subnets

https://gerrit.wikimedia.org/r/1198370

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2025, 10:31 AM
Jclark-ctr closed subtask T405560: Eqiad C/D refresh: 2 x test hosts for config validation as Resolved.Oct 24 2025, 4:56 PM
cmooney added a subtask: T409286: Nokia L3 bugs [Oct 2025].Nov 5 2025, 1:14 PM
Jclark-ctr closed subtask T396065: Eqiad: row C/D switch refresh cabling task as Resolved.Nov 19 2025, 2:04 PM
cmooney reopened subtask T405499: Remove lvs1018 L2 link to ssw1-e1-eqiad as Open.Nov 19 2025, 2:31 PM
VRiley-WMF changed the status of subtask T405499: Remove lvs1018 L2 link to ssw1-e1-eqiad from Open to In Progress.Dec 2 2025, 5:29 PM
VRiley-WMF closed subtask T405499: Remove lvs1018 L2 link to ssw1-e1-eqiad as Resolved.Dec 2 2025, 5:41 PM
cmooney closed subtask T405558: Nokia: add new switches in eqiad/codfw to monitoring and make 'active' as Resolved.Dec 10 2025, 8:33 PM
cmooney closed subtask T402588: Eqiad: row C/D switch refresh configuration task as Resolved.
cmooney closed subtask T404146: Netbox: General updates for Nokia switch support as Resolved.Dec 10 2025, 8:35 PM
Jclark-ctr moved this task from Backlog to High Priority Task on the ops-eqiad board.Dec 10 2025, 8:40 PM
Jclark-ctr moved this task from High Priority Task to Remote Work on the ops-eqiad board.
cmooney added a subtask: T412271: Decom Juniper EX/QFX switches in eqiad rows C/D.Dec 10 2025, 8:42 PM
Jclark-ctr closed subtask T412271: Decom Juniper EX/QFX switches in eqiad rows C/D as Resolved.Dec 15 2025, 3:22 PM
cmooney closed subtask T409286: Nokia L3 bugs [Oct 2025] as Resolved.Fri, Feb 6, 3:45 PM
cmooney mentioned this in T416872: Eqiad: move row-wide vlan gateways to Nokia switches.Mon, Feb 9, 1:19 PM
cmooney removed a subtask: T405562: Eqiad C/D refresh: move legacy switch uplinks to Nokias and migrate Vlan GWs.
cmooney removed a subtask: T405602: Eqiad row C/D switch refresh: LVS changes to support migration.
cmooney added a subtask: T405579: Eqiad C/D refresh: move asw2-c-eqiad CR uplinks to Nokia switches.Mon, Feb 9, 1:23 PM
cmooney added a subtask: Unknown Object (Task).
cmooney added a subtask: T409067: Eqiad C/D refresh: move asw2-d-eqiad CR uplinks to Nokia switches.
cmooney added a subtask: T405609: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad.
cmooney added a subtask: T405628: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad.
cmooney added a subtask: T410661: lvs1018: decom links to asw2-c2-eqiad and asw2-d7-eqiad.
cmooney removed a project: Infrastructure-Foundations.Mon, Feb 9, 1:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits