Page MenuHomePhabricator

eqsin downtime scheduling for switch upgrade
Closed, ResolvedPublic

Description

ops-eqsin has a planned switch upgrade, replacing the two older Juniper switches with new Nokia switches ordered via T417350 (cables) and T414840 (switches).

Jin was on-site to install these on 2026-07-24, but the rack PDUs and cabling are in the way of installation to the point that we need to schedule downtime for the site.

The work will involve:

  • Carefully unmounting the PDUs and twisting them out of the way to access the back rack rail.
  • Installation of new nokia switches asw1-60[34]-eqsin into U37 in rack 60[34]
  • wiring of new asw1-60[34]-eqsin to routers and msw[12].
  • corrective cable auditing and patching to bring site to standard

The work will not include:

  • migration of hosts from old to new switches, that will take place at a later date once new switches are fully online and accessible in their new permanent home.
  • migration will be scheduled for a later site visit after switch installation.

Request from traffic:

  • Scheduling and sign off to depool eqsin from serving traffic during the migration.
    • While Jin will make all attempts to not unseat any cables, having to unmount and move the PDU towers could very easily result in a power loss event. As such, DC Ops prefers to fully depool in advance of the work to avoid negative user impact.

Confirmed Schedule: 2026-07-29 @ 23:00 GMT to 2026-07-30 @ 03:00 GMT (July 30th Singapore AM, July 29th PM for US based folks)

Related Objects

Event Timeline

RobH triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
RobH updated the task description. (Show Details)

Sukhbir,

Please review the task description and provide feedback on when we could depool eqsin for the proposed work.

@ssingh I am willing to be available on my Monday evening/Tuesday Singapore time.

Thanks @BCornwall. @RobH: Is there a preference for the day of the week from your end? I am assuming we can't do tomorrow Singapore time since it is too short of a notice.

And do we know the expected maintenance window, even an approximate one?

Right now it is 13:55 GMT = 21:55 Singapore, so if we schedule now it won't be confirmed until Tuesday, for possibly a Wednesday appearance. GMT 00:00 = Singapore 08:00.

So Wednesday/Thursday/Friday for SG3 work would be Tuesday/Wednesday/Thursday evening for me.

With that in mind, if we approved today the work to depool SG3 for the work on Thursday Singapore AM would be Wednesday evening for me. That is ideal, as it gives us a day to review the directions and get a confirmation back from Jin.

Proposed window: 2026-07-30 @ 01:00 GMT (9AM Singapore). Depool to take place a few hours in advance (in case Jin can show up earlier, sometimes possible) so suggested depool is 2026-07-29 @ 23:00.

Would that work for folks?

Proposed window: 2026-07-30 @ 01:00 GMT (9AM Singapore). Depool to take place a few hours in advance (in case Jin can show up earlier, sometimes possible) so suggested depool is 2026-07-29 @ 23:00.

Would that work for folks?

That's fine by Traffic. @BCornwall or I can take it. So we are doing the depool at (please confirm):

Wednesday 2026-07-29 23:00 UTC / 16:00 Pacific / 19:00 Eastern

Do we know how long the work will take?

I'm going to estimate the work starting at 9AM Singapore will take up to 4 hours to complete, so we'd like the site depooled from 2026-07-29 @ 23:00 GMT to 2026-07-30 @ 03:00 GMT.

With that kind of timing, Is it possible to have traffic depool before Brett signs off for the evening on Thursday, and then someone in Traffic EU hours re-pool and monitor when their work day starts?

When the work is completed onsite I'll stay at the keyboard for 30 minutes to ensure no unexpected issues (it was 15 min but we introduced a spanning tree error last time that took 20 to show up.). I could also re-pool SG3 at that time, but it'll be pretty late for me so I prefer not to repool and monitor, since any caching issues will be out of scope for me to troubleshoot.

Thoughts?

I'm going to estimate the work starting at 9AM Singapore will take up to 4 hours to complete, so we'd like the site depooled from 2026-07-29 @ 23:00 GMT to 2026-07-30 @ 03:00 GMT.

Thanks, that sounds good.

With that kind of timing, Is it possible to have traffic depool before Brett signs off for the evening on Thursday, and then someone in Traffic EU hours re-pool and monitor when their work day starts?

We will coordinate this within Traffic but yes, it should not be on you to repool the site so we will make sure that someone from Traffic does it. That can either be Brett, or someone else in the EU time zone.

When the work is completed onsite I'll stay at the keyboard for 30 minutes to ensure no unexpected issues (it was 15 min but we introduced a spanning tree error last time that took 20 to show up.). I could also re-pool SG3 at that time, but it'll be pretty late for me so I prefer not to repool and monitor, since any caching issues will be out of scope for me to troubleshoot.

I think we can go ahead and confirm the maintenance event and I will update this task with who will depool and repool the site. Once you confirm to us here that the event is confirmed, I can send an email to sre-at-large@ as well with the details.

Thanks!

I've sent an email to Jin to confirm the date window of 2026-07-29 @ 23:00 GMT to 2026-07-30 @ 03:00 GMT, which is 2026-07-30 @ 09:00-13:00 Singapore time. It is past 10PM there now, so we likely won't get a confirmation from Jin until tomorrow.

Confirming I can do those times (2026-07-29 @ 16:00-20:00 PDT)

Jin confirmed via email just now for the window: 2026-07-29 @ 23:00 GMT to 2026-07-30 @ 03:00 GMT

I'll update our directions for him to reflect the changes in cadence and send over for his work.

@BCornwall Thanks for confirming!

Mentioned in SAL (#wikimedia-operations) [2026-07-29T22:20:24Z] <brett@cumin2002> START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: Switch upgrade maintenance window, T433097]

Mentioned in SAL (#wikimedia-operations) [2026-07-29T22:20:42Z] <brett@cumin2002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: Switch upgrade maintenance window, T433097]

Mentioned in SAL (#wikimedia-operations) [2026-07-30T01:38:28Z] <brett@cumin2002> START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: Switch upgrade maintenance window complete, T433097]

Mentioned in SAL (#wikimedia-operations) [2026-07-30T01:38:33Z] <brett@cumin2002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: Switch upgrade maintenance window complete, T433097]

Resolution Notes:

  • Work was completed without any unexpected downtime
  • The fibers/optics needed hadn't all been ordered, order being placed to fix via T417350
    • the new parts will arrive in advance of Jin's next visit
  • Jin's next visit will be to finish the fiber installation for the new switches, migrate hosts to the new switches, apply cable IDs to any cables missing them, and inventory our spare parts.

Thanks Brett and Papaul for working on this with us!