Page MenuHomePhabricator
Create Task
Maniphest T357547

☂️ Northward Datacentre Switchover (March 2024)
Closed, ResolvedPublic
Actions

Description

This is an umbrella ☂️ task for the upcoming Northward Switchover.

As of Sept 2023, switchovers take place at predictable dates; the work week of the Solar Equinox.

Important Dates:

Day 1 issues:

  • Kartotherian started running out of resources, so we had to repool kartotherian on codfw and restart the service on both datacentres
  • Thumbor was using swift.discovery.wmnet, thus thumbor on codfw was attempting to access swift on eqiad using codfw's creds, causing tons of 401s.
  • mw-on-k8s started working harder than usual, expected since we turned off multi-DC, we added some more resources just to be on the safe side. Specifically, we added 53 replicas to mw-web and 10 to mw-api-ext.
  • Unfortunate coincidence where around the services switchover, changeprop was overwhelmed for unrelated reasons, causing jobs to pile up

Day 2 issues:

  • While stopping all maintenance scripts (01-stop-maintenance), we found a user triggered script which we fiercely killed manually, and continued the process

Day 3 issues:

  • We switched to deploy1002.eqiad.wmnet without any issues.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Marostegui added a project: Datacenter-Switchover.Feb 22 2024, 10:42 AM
• Marostegui changed the status of subtask T358200: Database post-switchover tasks March 2024 from Open to Stalled.Feb 22 2024, 10:49 AM
akosiaris subscribed.Feb 22 2024, 11:30 AM
In T357547#9566904, @Marostegui wrote:

What is the idea? Will codfw remain depooled for a week or two? For DBAs this would be good so we can perform some maintenance in codfw.

As documented in https://wikitech.wikimedia.org/wiki/Switch_Datacenter/Recurring,_Equinox-based,_Data_Center_Switchovers#Overview.

For the next 7 calendar days after the read-only phase of the Switchover, traffic will be flowing solely to one of the 2 data centers, effectively rendering the other data center inactive.
On the Wednesday following the read-only phase of the Switchover, that is right after exactly 7 days, traffic will start flowing, in the normal Multi-DC way, to both data centers.

This has been done for the September 2023 Switchover and will be done for this one too.

We do have an open item to see if it makes sense to limit this from 7 days to 5 by repooling in Multi-DC for the duration of the weekend, but this is best handled in a different task and at the proper timing.

• Marostegui added a comment.Feb 22 2024, 12:47 PM

I'd love if it can be a bit longer than 7 days as we can do lots of operational maintenance and save a bunch of time, but anyway, to be discussed once the time is approaching.
Thanks Alex

jijiki renamed this task from Northward Datacentre Switchover (March 2024) to ☂️ Northward Datacentre Switchover (March 2024) .Feb 22 2024, 3:04 PM
jijiki updated the task description. (Show Details)
jijiki added a subtask: T358233: MoveComms support for Northward Datacentre Switchover (March 2024).Feb 22 2024, 3:42 PM
jijiki updated the task description. (Show Details)
ABran-WMF subscribed.Feb 23 2024, 7:51 AM
joanna_borun removed projects: Infrastructure-Foundations, netops.Feb 26 2024, 3:34 PM
LSobanski moved this task from Incoming to Consultation on the collaboration-services board.Feb 27 2024, 12:09 PM
LSobanski added a subtask: T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024.
Trizek-WMF changed the status of subtask T358233: MoveComms support for Northward Datacentre Switchover (March 2024) from Open to In Progress.Feb 29 2024, 2:37 PM
RLazarus added a subtask: T359130: Update DC switchover cookbooks to handle maintenance scripts on k8s.Mar 5 2024, 1:46 AM
herron subscribed.Mar 5 2024, 8:23 PM
gerritbot added a comment.Mar 9 2024, 10:52 PM

Change 1009854 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/cookbooks@master] sre.switchdc.mediawiki: update descriptions

https://gerrit.wikimedia.org/r/1009854

gerritbot added a project: Patch-For-Review.Mar 9 2024, 10:52 PM
RLazarus removed a subtask: T359130: Update DC switchover cookbooks to handle maintenance scripts on k8s.Mar 11 2024, 9:36 PM
RLazarus subscribed.Mar 11 2024, 9:38 PM

FYI: I redid a dry run and live test for 01-stop-maintenance.py after https://gerrit.wikimedia.org/r/1008583 and it's good to go.

• Marostegui closed subtask T358199: Database pre-switchover tasks March 2024 as Resolved.Mar 14 2024, 9:50 AM
ABran-WMF mentioned this in T360116: Upgrade s5 to MariaDB 10.6.Mar 14 2024, 1:58 PM
ABran-WMF mentioned this in T360332: Make the cupe_actor column nullable on WMF wikis.Mar 19 2024, 7:52 AM
jijiki closed subtask T359154: Update DC switchover cookbooks to handle mw-jobrunners as Resolved.Mar 19 2024, 1:02 PM
gerritbot added a comment.Mar 19 2024, 1:15 PM

Change 1012645 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] traffic: Completely depool codfw from user traffic (switchover #1)

https://gerrit.wikimedia.org/r/1012645

gerritbot added a comment.Mar 19 2024, 1:43 PM

Change 1009854 merged by jenkins-bot:

[operations/cookbooks@master] sre.switchdc.mediawiki: update descriptions

https://gerrit.wikimedia.org/r/1009854

gerritbot added a comment.Mar 19 2024, 2:04 PM

Change 1012645 merged by Effie Mouzeli:

[operations/dns@master] traffic: Completely depool codfw from user traffic (switchover #1)

https://gerrit.wikimedia.org/r/1012645

Stashbot added a comment.Mar 19 2024, 2:07 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-19T14:07:25Z] <effie> Completely depool codfw from user traffic - T357547

ops-monitoring-bot added a comment.Mar 19 2024, 2:16 PM

jiji@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter depool all services in codfw: Northward DC Switchover, March 2024 - T357547 started.

Stashbot added a comment.Mar 19 2024, 2:16 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-19T14:16:55Z] <jiji@cumin1002> START - Cookbook sre.discovery.datacenter depool all services in codfw: Northward DC Switchover, March 2024 - T357547

Stashbot added a comment.Mar 19 2024, 2:22 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-19T14:22:12Z] <effie> depooling services from codfw - T357547

Maintenance_bot removed a project: Patch-For-Review.Mar 19 2024, 2:31 PM
ops-monitoring-bot added a comment.Mar 19 2024, 2:40 PM

jiji@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter depool all services in codfw: Northward DC Switchover, March 2024 - T357547 completed.

Stashbot added a comment.Mar 19 2024, 2:40 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-19T14:40:03Z] <jiji@cumin1002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in codfw: Northward DC Switchover, March 2024 - T357547

gerritbot added a comment.Mar 19 2024, 2:40 PM

Change 1012686 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mw-on-k8s: raise replicas for add ro traffic

https://gerrit.wikimedia.org/r/1012686

gerritbot added a project: Patch-For-Review.Mar 19 2024, 2:41 PM
Stashbot added a comment.Mar 19 2024, 3:17 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-19T15:17:27Z] <claime> Raising mw-web and mw-api-ext replicas for additional read-only traffic - T357547

gerritbot added a comment.Mar 19 2024, 3:17 PM

Change 1012686 merged by jenkins-bot:

[operations/deployment-charts@master] mw-on-k8s: raise replicas for add ro traffic

https://gerrit.wikimedia.org/r/1012686

Maintenance_bot removed a project: Patch-For-Review.Mar 19 2024, 3:30 PM
gerritbot added a comment.Mar 19 2024, 3:32 PM

Change 1012698 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mw-web: Bump replicas another 15%

https://gerrit.wikimedia.org/r/1012698

gerritbot added a project: Patch-For-Review.Mar 19 2024, 3:32 PM
gerritbot added a comment.Mar 19 2024, 3:35 PM

Change 1012698 merged by jenkins-bot:

[operations/deployment-charts@master] mw-web: Bump replicas another 15%

https://gerrit.wikimedia.org/r/1012698

akosiaris added a comment.Mar 19 2024, 3:55 PM

We had to repool kartotherian in codfw as we had a CPU exhaustion event in eqiad right after the services switchover. Since some kartotherian endpoints create an amplification effect to kartotherian itself, we opted for restarting kartotherian in eqiad to fix that.

gerritbot added a comment.Mar 19 2024, 4:04 PM

Change 1012706 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/alerts@master] mw-on-k8s: Lower idle %age for saturation alert

https://gerrit.wikimedia.org/r/1012706

Clement_Goubert added a comment.EditedMar 19 2024, 4:11 PM

Some tweaking of replicas size was needed on mw-on-k8s, which was expected as this is the first switchover where more of the external traffic goes to it than to bare-metal clusters.

gerritbot added a comment.Mar 19 2024, 4:17 PM

Change 1012706 merged by jenkins-bot:

[operations/alerts@master] mw-on-k8s: Lower idle %age for saturation alert

https://gerrit.wikimedia.org/r/1012706

Maintenance_bot removed a project: Patch-For-Review.Mar 19 2024, 4:30 PM
gerritbot added a comment.Mar 20 2024, 9:51 AM

Change 1013005 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] wmnet: Update DNS records for master dbs to eqiad (switchover #2)

https://gerrit.wikimedia.org/r/1013005

gerritbot added a project: Patch-For-Review.Mar 20 2024, 9:51 AM
MatthewVernon subscribed.Mar 20 2024, 1:42 PM

Noting here for future reference - we found that thumbor was incorrectly using the global discovery record for swift, which meant that codfw-thumbor was trying to talk to eqiad-swift after codfw-swift was depooled, resulting in a rise in TempAuth errors (and 401s):

swift_tempauth_graph.png (812×1 px, 91 KB)

This was fixed via (initially repooling codfw swift and) https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1013033

Stashbot added a comment.Mar 20 2024, 1:48 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T13:48:50Z] <jiji@deploy2002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover - T357547

Stashbot added a comment.Mar 20 2024, 2:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T14:18:44Z] <marostegui> Test write T357547

gerritbot added a comment.Mar 20 2024, 2:37 PM

Change 1013005 merged by Effie Mouzeli:

[operations/dns@master] DBs: Update DNS records for master DBs to eqiad (switchover #2)

https://gerrit.wikimedia.org/r/1013005

gerritbot added a comment.Mar 20 2024, 2:44 PM

Change 1013064 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] maintenance: Update DNS records for maintenance host (switchover #3)

https://gerrit.wikimedia.org/r/1013064

gerritbot added a comment.Mar 20 2024, 2:49 PM

Change 1013064 merged by Effie Mouzeli:

[operations/dns@master] maintenance: Update DNS records for maintenance host (switchover #3)

https://gerrit.wikimedia.org/r/1013064

Stashbot added a comment.Mar 20 2024, 2:50 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T14:50:18Z] <jiji@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover - T357547 (duration: 61m 28s)

• Marostegui changed the status of subtask T358200: Database post-switchover tasks March 2024 from Stalled to Open.Mar 20 2024, 3:07 PM
gerritbot added a comment.Mar 20 2024, 3:09 PM

Change 1013070 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] geo-maps: make eqiad the default datacentre (switchover #4)

https://gerrit.wikimedia.org/r/1013070

gerritbot added a comment.Mar 20 2024, 3:21 PM

Change 1013070 merged by Effie Mouzeli:

[operations/dns@master] geo-maps: make eqiad the default datacentre (switchover #4)

https://gerrit.wikimedia.org/r/1013070

gerritbot added a comment.Mar 20 2024, 3:28 PM

Change 1013083 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/mediawiki-config@master] debug.json: List primary DC servers first (switchover #5)

https://gerrit.wikimedia.org/r/1013083

jijiki updated the task description. (Show Details)Mar 20 2024, 3:48 PM
jijiki updated the task description. (Show Details)
jijiki updated the task description. (Show Details)
gerritbot added a comment.Mar 20 2024, 3:57 PM

Change 1013083 merged by jenkins-bot:

[operations/mediawiki-config@master] debug.json: List primary DC servers first (switchover #5)

https://gerrit.wikimedia.org/r/1013083

Stashbot added a comment.Mar 20 2024, 3:58 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:58:14Z] <jiji@deploy2002> Started scap: Backport for [[gerrit:1013083|debug.json: List primary DC servers first (switchover #5) (T357547)]]

Stashbot added a comment.Mar 20 2024, 4:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T16:03:34Z] <jiji@deploy2002> jiji: Backport for [[gerrit:1013083|debug.json: List primary DC servers first (switchover #5) (T357547)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

• Marostegui closed subtask T358200: Database post-switchover tasks March 2024 as Resolved.Mar 20 2024, 4:08 PM
Stashbot added a comment.Mar 20 2024, 4:22 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T16:22:41Z] <jiji@deploy2002> Finished scap: Backport for [[gerrit:1013083|debug.json: List primary DC servers first (switchover #5) (T357547)]] (duration: 24m 27s)

Trizek-WMF closed subtask T358233: MoveComms support for Northward Datacentre Switchover (March 2024) as Resolved.Mar 20 2024, 4:29 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 20 2024, 4:30 PM
gerritbot added a comment.Mar 21 2024, 11:53 AM

Change 1013272 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] deployment: update deployment DNS record to deploy1002 (switchover #6)

https://gerrit.wikimedia.org/r/1013272

gerritbot added a project: Patch-For-Review.Mar 21 2024, 11:53 AM
gerritbot added a comment.Mar 21 2024, 11:57 AM

Change 1013274 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] hieradata: update deployment_server to deploy1002 (switchover #7)

https://gerrit.wikimedia.org/r/1013274

gerritbot added a comment.Mar 21 2024, 12:06 PM

Change 1013272 merged by Effie Mouzeli:

[operations/dns@master] deployment: update deployment DNS record to deploy1002 (switchover #6)

https://gerrit.wikimedia.org/r/1013272

gerritbot added a comment.Mar 21 2024, 12:08 PM

Change 1013274 merged by Effie Mouzeli:

[operations/puppet@production] hieradata: update deployment_server to deploy1002 (switchover #7)

https://gerrit.wikimedia.org/r/1013274

Maintenance_bot removed a project: Patch-For-Review.Mar 21 2024, 12:30 PM
Dzahn mentioned this in T360623: SystemdUnitFailed - releases2003.Mar 21 2024, 4:16 PM
jijiki updated the task description. (Show Details)Mar 21 2024, 4:21 PM
jijiki updated the task description. (Show Details)
Clement_Goubert mentioned this in T360763: Move 70% of mediawiki external requests to mw on k8s.Mar 22 2024, 11:23 AM
jijiki updated the task description. (Show Details)Mar 26 2024, 4:18 PM
gerritbot added a comment.Mar 27 2024, 2:02 PM

Change #1015037 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] traffic: Pool codfw for user traffic (switchover #8)

https://gerrit.wikimedia.org/r/1015037

gerritbot added a project: Patch-For-Review.Mar 27 2024, 2:02 PM
Stashbot added a comment.Mar 27 2024, 2:19 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-27T14:19:08Z] <effie> Day 8: Pool active/active services on codfw - T357547

ops-monitoring-bot added a comment.Mar 27 2024, 2:21 PM

jiji@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in codfw: Pool active/active services on codfw - T357547 started.

Stashbot added a comment.Mar 27 2024, 2:21 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-27T14:21:28Z] <jiji@cumin1002> START - Cookbook sre.discovery.datacenter pool all active/active services in codfw: Pool active/active services on codfw - T357547

ops-monitoring-bot added a comment.Mar 27 2024, 2:40 PM

jiji@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in codfw: Pool active/active services on codfw - T357547 completed.

Stashbot added a comment.Mar 27 2024, 2:41 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-27T14:40:59Z] <jiji@cumin1002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in codfw: Pool active/active services on codfw - T357547

Stashbot added a comment.Mar 27 2024, 2:45 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-27T14:45:32Z] <effie> Day 8: Pool codfw for user traffic - T357547

gerritbot added a comment.Mar 27 2024, 2:46 PM

Change #1015037 merged by Effie Mouzeli:

[operations/dns@master] traffic: Pool codfw for user traffic (switchover #8)

https://gerrit.wikimedia.org/r/1015037

jijiki added a subtask: T361113: Improve readability of Switchover documentation.Mar 27 2024, 2:59 PM
jijiki closed subtask T358286: SRE comms for Northward Datacentre Switchover (March 2024) as Resolved.Mar 27 2024, 3:42 PM
jijiki closed this task as Resolved.Mar 27 2024, 3:44 PM
jijiki claimed this task.

Switchover is done, it is Day 8, and we are back to Multi-DC. Thank you serviceops and @akosiaris for being good teammates and keeping an eye on things.

lmata moved this task from Up next to Done on the SRE Observability (FY2023/2024-Q3) board.Thu, Apr 4, 6:10 PM
eoghan closed subtask T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024 as Resolved.Fri, Apr 12, 1:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL