Page MenuHomePhabricator

MoveComms support for Northward Datacentre Switchover (March 2026; codfw to eqiad)
Closed, ResolvedPublic

Description

Dear MoveComms,

We are planning a datacentre switchover for the week of March 24th (week 13) with the following schedule:

The expected impact is 2-3 minutes of read-only on Wednesday, 25 March 2026 @ 15:00 UTC.

We continue to follow the process described in Recurring, Equinox-based, Data Center Switchovers, in particular:

  • There is no switchback! We are staying in eqiad until the next switchover.
  • Future switchovers are predictable and take place every 6 months; always on the week of an equinox.

Let ServiceOps new know if you need more info on the changes.

Thank you!


Planning

**As soon as the task is received by Movement Communications ==

Three weeks before

  • Tech News message (initial warning in this week's issue, reuse this message) - @STei-WMF
  • Update the message to communities
    • Check on dates and links
    • Edit qqq messages with the new Zonestamp link
  • Have the message being translated by emailing both:
    • translators-l for translations
      • Mention Daylight saving time for this message - not needed
    • wikitech-ambassadors for information and translations
  • Monitor the message's talk page

Two weeks before

The week before

    • Messaging:
      • wikitech-ambassadors and translators-l once again (reminders)
      • wmfall (done by SRE)
      • WMF Slack (done by SRE)
    • Added to the news on the Meta front page
    • Send the message to communities
  • Tech News ("this is happening this week" in this week's issue)
  • Agree with SRE on who will inform Movement Communication of the advancement and success of the process.

The week it happens

  • monitor the wikis
  • when the read-only is done and confirmed by SRE's spokeperson, remove the banner.

The week after, maximum

  • Debrief how it went, and document (comment on this task)

Event Timeline

Trizek-WMF updated the task description. (Show Details)
Trizek-WMF added a subscriber: STei-WMF.

Hi @Blake, I'll be your MovComms support for this. Any notable changes in the process since the last switch?

Trizek-WMF changed the task status from Open to In Progress.Mar 6 2026, 1:50 PM
Trizek-WMF updated the task description. (Show Details)

Hello! There have been no notable changes since the last switchover.

@STei-WMF, can I let you do all announcement in Tech News ? Thanks!

@Blake, can you tell me who will inform Movement Communication (in this case, me) of the advancement and success of the process? I need to know so that I can breathe. :p If the process goes longer, it is important to keep us informed so that we can reassure community members who'd ask for an update.

@STei-WMF, can I let you do all announcement in Tech News ? Thanks!

Covered.

Hi @Trizek-WMF,

Any significant updates will be tracked on the Phabricator task for the switchover, which is T413974. If it would be helpful, I can also update this task with progress - I expect the biggest update will be on Wednesday the 25th, when we leave read-only, and have mostly cleanup left to do. If there are any changes with respect to duration, or problems that arise as a result of the switchover, I'll update T413974, and will reach out. If there's anything else that would be helpful, please let me know. Cheers!

Thank you Blake.

Only the part that affects wikis' editing (on the 25th) concerns me. :)

I subscribed to T413974 and, on the 25th, I'll remain available on Slack if you need.

If you plan to postpone the switchover for any reason, please use this task (if decided a few days before) of Slack (if decided on the 25th) as I'll need to update a few community messages.

@Trizek-WMF, what is the 'wmfall' messaging task referenced in the description? What that is and how to do it are not currently in the SRE documentation for the switchover. Thanks!

@Trizek-WMF, what is the 'wmfall' messaging task referenced in the description? What that is and how to do it are not currently in the SRE documentation for the switchover. Thanks!

It was the general mailing list, but it seems that SRE switched it to a message on the #general Slack channel.

Ah, great, thanks very much.

Send the message to communities

I skipped this part. Many colleagues questioned the relevance of the MassMessage as we already used Tech News to inform communities. Messaging fatigue is real, some wikis only have old announcements at their community boards. Let's see how it goes without messaging.

Switchover activities are complete - thanks!

Post-switch report

Not posting the big message at the wikis was apparently okay. Tech News announcements look enough, as the recurrence of the switchover around equinoxes seems to be a custom now. I'll update the Movement Comms documentation accordingly.

We got this report from @ValerioBozz:

On itwiki the kind user @Ensahequ: reported that during the server switch the user was logged-out or with session inconsistencies (with / without their name on the top bar). If this is something known, maybe it's a symptom which could be mentioned in the next similar announcements. If this is something that should not happen instead... well I don't know :D lol --ValerioBozz

I think that it is coincidental, more likely related to T383566: Some editors are frequently getting logged out (multiple times a day) (and past T372702: editors are repeatedly getting logged out (August 2024)), but I'm documenting it here to get a proper confirmation. :)

@Trizek-WMF After a chat with some folks on the team, it sounds unlikely that this is related to the switchover, but I'll make a note to follow up and see if we encounter a similar issue next iteration. Thanks for all your help!