Page MenuHomePhabricator

FY2526 Q3 ulsfo: switch refresh
Closed, ResolvedPublic

Description

This task will track the refresh of the productions switches in ULSFO. We will be replacing the old switches (Juniper) with Nokia switches. We will also be changing the configuration from Virtual chassis to flat BGP setup between the core routers and the switches.

Scheduling

  • Stage 1 - switch racking and configuration
    • Remote hands follows directions (submitted via portal by Rob) for racking and cabling the two new switches to allow remote connectivity and setup.
  • Stage 2 - host migrations
    • Once switches are fully online and ready, remote hands task will be filed to migrate the ports of each host 1 by 1 from old to new switches.
    • new switches test online and functional
    • old switches are wiped by netops
  • Stage 3 - cleanup and old hardware removal
    • once all hosts are confirmed online and operational on the new switches the old switches will be wiped to factory defaults and disconnected from all cables for future recycling.

Sub Tasks

T408511
T408892

Hardware

Switches and optics/fibers on site.

Switches: myriad order : T398503 : Shipping Ticket 01110451 stored in Digital Realty NOC storage.
Optics/Fibers : fs.com order : T408652 : Shipping Ticket 01118256 stored in Digital Realty NOC storage.

Remote Hands Directions

In progress via this google doc.

Details

Other Assignee
RobH
Related Changes in Gerrit:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Papaul triaged this task as Medium priority.Oct 28 2025, 6:43 AM
Papaul added a subtask: Unknown Object (Task).Oct 29 2025, 12:05 AM
RobH closed subtask Unknown Object (Task) as Resolved.Nov 18 2025, 7:54 PM
ayounsi added a subtask: Unknown Object (Task).Dec 17 2025, 3:00 PM
RobH closed subtask Unknown Object (Task) as Resolved.Jan 7 2026, 11:16 PM

Mentioned in SAL (#wikimedia-operations) [2026-01-14T15:43:27Z] <cdobbins@cumin1003> START - Cookbook sre.dns.admin DNS admin: depool site ulsfo [reason: switch work, T408510]

Mentioned in SAL (#wikimedia-operations) [2026-01-14T15:43:36Z] <cdobbins@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site ulsfo [reason: switch work, T408510]

Mentioned in SAL (#wikimedia-operations) [2026-01-14T17:26:34Z] <sukhe@cumin1003> START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4010*} and A:liberica (T408510)

Mentioned in SAL (#wikimedia-operations) [2026-01-14T17:26:53Z] <sukhe@cumin1003> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4010*} and A:liberica (T408510)

Mentioned in SAL (#wikimedia-operations) [2026-01-14T17:28:56Z] <sukhe@cumin1003> START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4009*} and A:liberica (T408510)

Mentioned in SAL (#wikimedia-operations) [2026-01-14T17:29:14Z] <sukhe@cumin1003> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4009*} and A:liberica (T408510)

Mentioned in SAL (#wikimedia-operations) [2026-01-14T17:29:35Z] <sukhe@cumin1003> START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4008*} and A:liberica (T408510)

Mentioned in SAL (#wikimedia-operations) [2026-01-14T17:29:54Z] <sukhe@cumin1003> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4008*} and A:liberica (T408510)

Mentioned in SAL (#wikimedia-operations) [2026-01-14T18:05:24Z] <cdobbins@cumin1003> START - Cookbook sre.dns.admin DNS admin: pool site ulsfo [reason: switch work, T408510]

Mentioned in SAL (#wikimedia-operations) [2026-01-14T18:05:36Z] <cdobbins@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site ulsfo [reason: switch work, T408510]

@Papaul: Hi. Have we decided on a date for this? Asking for Traffic planning purpose, thanks!

@ssingh Hey unfortunately we don't have a date yet because the switches are not racked and we haven't done the initial configuration.. I will get back with you with an ETA as soon as I can. Thank you.

@ssingh Hey unfortunately we don't have a date yet because the switches are not racked and we haven't done the initial configuration.. I will get back with you with an ETA as soon as I can. Thank you.

No worries, thanks Papaul. We have allocated time for this so whenever you are ready, just let us know in this task.

RobH mentioned this in Unknown Object (Task).Feb 17 2026, 4:07 PM

Change #1240188 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Update mr1-ulsfo IP, remove eqiad old rows C/D

https://gerrit.wikimedia.org/r/1240188

Change #1240188 merged by Ayounsi:

[operations/puppet@production] Update mr1-ulsfo IP, remove eqiad old rows C/D

https://gerrit.wikimedia.org/r/1240188

RobH renamed this task from ULSFO: switch refresh to FY2526 Q3 ulsfo: switch refresh.Feb 18 2026, 5:00 PM
RobH updated Other Assignee, added: RobH.
RobH added a subtask: Unknown Object (Task).Thu, May 14, 2:42 PM

The ULSFO switch refresh is complete. Good to close this task.