Page MenuHomePhabricator

Southward Datacenter Switchover (September 2024)
Closed, ResolvedPublic

Description

This is the umbrella task for the September 2024 Southward Datacenter Switchover (eqiad to codfw).

Please file tasks for related work as subtasks.

Important Dates:

Note: The above times are all 15:00 UTC (i.e., 1h later than the 14:00 UTC target of the prior switchover).

Previous switchover (March 2024): T357547

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Calling to attention T375382: Post pc1013 crash, failover may need to be done by DBA before the switchover.

@Dreamy_Jazz - ack, thank you!

@Clement_Goubert - thanks for flagging! pc1015 has now been swapped in for pc3, so this should be stable now.

The remaining complexity is T375345: ulsfo is depooled due to issues with cr3-ulsfo. The latter is in a bad state (recurring every few hours) and we're waiting on input from JTAC on how to proceed.

This raises the question of whether we're comfortable depooling eqiad tomorrow at 15:00 if ulsfo is either still depooled or at-risk of being depooled again (if the device has further issues after a reboot).

In part, this reflects our confidence in whether codfw can survive being the only pooled cache location in NA from a hardware and network ingress / egress perspective, as well as our desire to be "voluntarily" SPoF on a single site for NA.

Per discussion in #wikimedia-traffic, the former should be fine, but this is of course not a desirable state. As long as we retain the ability to re-pool eqiad if needed, that's probably sufficient - i.e., defer any disruptive maintenance we had previously planned under cover of the switchover that would risk being unable to rapidly put it back in service.

In any case, we'll revisit this early tomorrow and make the call then, and have a number of options as to how to proceed. FYI @ssingh @ayounsi

Change #1075056 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] mw-(api-ext|web): scale back to 75% at p95 targets

https://gerrit.wikimedia.org/r/1075056

Change #1075056 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-ext|web): scale back to 75% at p95 targets

https://gerrit.wikimedia.org/r/1075056

Mentioned in SAL (#wikimedia-operations) [2024-09-24T14:19:23Z] <swfrench-wmf> scaled up mw-api-ext ahead of traffic+services switchover - T370962

Mentioned in SAL (#wikimedia-operations) [2024-09-24T14:21:47Z] <swfrench-wmf> scaled up mw-web ahead of traffic+services switchover - T370962

Mentioned in SAL (#wikimedia-operations) [2024-09-24T15:01:11Z] <swfrench-wmf> starting switchover day 1 - T370962

Mentioned in SAL (#wikimedia-operations) [2024-09-24T15:01:52Z] <swfrench@cumin1002> START - Cookbook sre.dns.admin DNS admin: depool site eqiad [reason: Datacenter Switchover, T370962]

Mentioned in SAL (#wikimedia-operations) [2024-09-24T15:02:12Z] <swfrench@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqiad [reason: Datacenter Switchover, T370962]

swfrench@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover - T370962 started.

Mentioned in SAL (#wikimedia-operations) [2024-09-24T15:23:42Z] <swfrench@cumin1002> START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover - T370962

swfrench@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover - T370962 completed.

Mentioned in SAL (#wikimedia-operations) [2024-09-24T15:49:12Z] <swfrench@cumin1002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter Switchover - T370962

Mentioned in SAL (#wikimedia-operations) [2024-09-24T15:50:17Z] <swfrench-wmf> switchover day 1 actions are complete - T370962

Mentioned in SAL (#wikimedia-operations) [2024-09-24T16:06:59Z] <swfrench-wmf> repooled swift-ro in eqiad to potentially mitigate issues with thumbor - T370962

Mentioned in SAL (#wikimedia-operations) [2024-09-24T16:18:27Z] <swfrench-wmf> repooled swift in eqiad to potentially mitigate issues with thumbor and swift - T370962

Mentioned in SAL (#wikimedia-operations) [2024-09-24T16:38:58Z] <swfrench-wmf> switched logstash.discovery.wmnet back to eqiad due to reports of stale dashboards - T370962

The swift-https service has three associated discovery services:

  • swift (A/A)
  • swift-rw (A/A)
  • swift-ro (A/P)

This is in addition to the canonical ms-fe.svc.{site}.wmnet addresses used by, e.g., the collocated thumbors.

At least as far as codesearch is aware, there appear to be no active uses of swift-ro/rw at this time (consistent with the # TODO: remove this from DNS! comments in the service catalog).

What we suspect happened here is that upon depooling swift at eqiad (which backs upload.wm.o [0]) at 15:35:34, thumbor in codfw became overloaded (i.e., could not sustain full global load).

This was mitigated by repooling swift in eqiad (though initially only swift-ro was repooled there, due to some confusion as to the distinction between the services).

In any case, we should consider some subset of the following:

  1. Adding capacity to thumbor and again depooling eqiad - this may not be advisable at the moment, given the resources involved (14 CPU-s/s per pod, already at 60 replicas).
  2. Adding a note to the Switchover Day 1 documentation to be aware of this potential issue, and / or adding an EXCLUDED_SERVICES opt-out in the sre.discovery.datacenter cookbook.
  3. Retiring the seemingly unused discovery services to avoid future confusion.

[0] https://gerrit.wikimedia.org/g/operations/puppet/+/89b939712de23ebeae2313d57cb639933b6e81e5/hieradata/common/profile/trafficserver/backend.yaml#337

Mentioned in SAL (#wikimedia-operations) [2024-09-25T13:50:08Z] <swfrench-wmf> kartotherian repooled in eqiad due load issues - T370962

Mentioned in SAL (#wikimedia-operations) [2024-09-25T14:26:35Z] <swfrench@deploy1003> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover - T370962

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:00:02.369386

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:00:11.432680

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:05:39.789229

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw - finished with status: FAILURE elapsed time: 0:00:11.045356

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:00:26.491358

Mentioned in SAL (#wikimedia-operations) [2024-09-25T14:54:12Z] <swfrench-wmf> reset failed units on mwmaint1002 to unblock 01-stop-maintenance - T370962

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw - MediaWiki read-only period starts at: 2024-09-25 14:58:42.440378

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:00:27.715698

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:00:34.600122

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:00:38.463642

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:00:03.071019

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw - MediaWiki read-only period ends at: 2024-09-25 15:01:28.078892

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:00:05.699805

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:00:47.559635

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:02:24.880856

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:00:38.092436

Change #1073897 merged by Scott French:

[operations/dns@master] wmnet: update CNAME records for DB masters to codfw

https://gerrit.wikimedia.org/r/1073897

swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqiad to codfw - finished with status: SUCCESS elapsed time: 0:09:52.639538

Mentioned in SAL (#wikimedia-operations) [2024-09-25T15:24:50Z] <swfrench@deploy1003> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover - T370962 (duration: 58m 14s)

Change #1073898 merged by Scott French:

[operations/dns@master] wmnet: update CNAME record for maintenance host to codfw

https://gerrit.wikimedia.org/r/1073898

Change #1073899 merged by Scott French:

[operations/dns@master] geo-maps: update map default to list codfw first

https://gerrit.wikimedia.org/r/1073899

Change #1073895 merged by jenkins-bot:

[operations/mediawiki-config@master] debug.json: order codfw (primary) DC backends first

https://gerrit.wikimedia.org/r/1073895

Mentioned in SAL (#wikimedia-operations) [2024-09-25T15:48:13Z] <swfrench@deploy1003> Started scap sync-world: Backport for [[gerrit:1073895|debug.json: order codfw (primary) DC backends first (T370962)]]

Mentioned in SAL (#wikimedia-operations) [2024-09-25T15:50:35Z] <swfrench@deploy1003> swfrench: Backport for [[gerrit:1073895|debug.json: order codfw (primary) DC backends first (T370962)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-09-25T15:56:19Z] <swfrench@deploy1003> Finished scap sync-world: Backport for [[gerrit:1073895|debug.json: order codfw (primary) DC backends first (T370962)]] (duration: 08m 05s)

Mentioned in SAL (#wikimedia-operations) [2024-09-25T15:57:03Z] <swfrench-wmf> switchover day 2 manual steps are complete - T370962

Change #1075625 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/cookbooks@master] sre.discovery.datacenter: exclude kartotherian-ssl

https://gerrit.wikimedia.org/r/1075625

Mentioned in SAL (#wikimedia-operations) [2024-09-26T00:41:04Z] <sukhe@cumin1002> START - Cookbook sre.dns.admin DNS admin: pool site eqiad [reason: repooling due to repeated port utilization alerts, T370962]

Mentioned in SAL (#wikimedia-operations) [2024-09-26T00:41:12Z] <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site eqiad [reason: repooling due to repeated port utilization alerts, T370962]

Cross-posting / paraphrasing from IRC:

This afternoon, we received some outbound port utilization alerts for cr2-codfw, due to modest spikes in egress on top of the already large baseline increase with ulsfo and eqiad depooled.

Given that this was likely to repeat throughout the next few hours (and on subsequent days around the same time) we made the decision to repool eqiad for edge traffic (done at 00:41 - see above).

One potential concern we discussed was stress on eqiad <> codfw transport links while A/A services are depooled in eqiad. However, given that swift was repooled there yesterday (see above), we considered this unlikely (i.e., upload.w.o origin stays in eqiad).

With eqiad repooled, egress in codfw has been greatly reduced, and transport links (particularly the currently preferred cr1-codfw:et-1/0/2 - cr1-eqiad:et-1/1/2 link) look fine.

We should regroup on Thursday to consider whether it makes sense to (again) electively depool eqiad given the current state and what would be necessary to enable that.

Mentioned in SAL (#wikimedia-operations) [2024-09-26T15:01:30Z] <swfrench-wmf> starting switchover day 3 deployment server switch - T370962

Change #1073900 merged by Scott French:

[operations/dns@master] wmnet: update deployment CNAME record to deploy2002

https://gerrit.wikimedia.org/r/1073900

Mentioned in SAL (#wikimedia-operations) [2024-09-26T15:04:40Z] <swfrench-wmf> running authdns-update for deployment CNAME switch - T370962

Change #1073894 merged by Scott French:

[operations/puppet@production] hieradata: update deployment_server to deploy2002

https://gerrit.wikimedia.org/r/1073894

Mentioned in SAL (#wikimedia-operations) [2024-09-26T15:34:17Z] <swfrench@deploy2002> Started scap sync-world: No-op deployment to verify switchover - T370962

Mentioned in SAL (#wikimedia-operations) [2024-09-26T16:04:10Z] <swfrench@deploy2002> Finished scap sync-world: No-op deployment to verify switchover - T370962 (duration: 29m 53s)

Mentioned in SAL (#wikimedia-operations) [2024-09-26T16:27:37Z] <swfrench-wmf> done with switchover day 3 deployment server switch - T370962

Mentioned in SAL (#wikimedia-operations) [2024-09-26T17:08:23Z] <swfrench-wmf> cleared contents of a stale scap dsh group resource on deploy2002 - T370962

Change #1075981 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] scap: remove stale production dsh groups

https://gerrit.wikimedia.org/r/1075981

Quick explanation of T370962#10180675 (stale scap dsh groups):

When running a test scap sync-world after switching the active deployment server to deploy2002, everything seems to have worked as expected with the exception of a non-fatal error logged during the php-fpm-restarts stage: P69429.

Upon looking more closely, it looks like deploy1003 (prior active) and deploy2002 (new active) had different content for the no-longer-relevant parsoid_php dsh group, with the latter still containing both parse1001 and parse2001 (and the former containing nothing).

In any case, I believe this is the result of a race after https://gerrit.wikimedia.org/r/1058169 was merged: that removes both the node objects from etcd and (transitively) the definitions of the dsh group puppet resources.

  • If puppet runs on the deployment host before confd notices deletion of the keys from etcd (runs with a 5m interval here), the dsh group file will be abandoned with the old parsoid hosts still listed.
  • If confd runs first (as was presumably the case on deploy1003), the group file is abandoned "empty" (i.e., template comments, but no hosts).

In any case, I removed the parsoid hosts from the stale group file manually, and https://gerrit.wikimedia.org/r/1075981 should prevent scap from even considering them.

Collecting remaining follow-ups so far:

kartotherian: Given that we've seen capacity related failures during the last two switchovers, it should (at least until we have the option scaling its capacity on-demand on k8s) be opted out of the day 1 services switchover. This is pending in https://gerrit.wikimedia.org/r/1075625.

thumbor / swift: Some subset of the actions described in T370962#10172473 should be adopted. I would propose that:

  • Short term: We similarly add an opt-out for the swift-https (i.e., swift.discovery.wmnet) (https://gerrit.wikimedia.org/r/1077111) and prioritize removing the swift-r[ow] discovery services to avoid future confusion (T376237).
  • Medium term: We need a better understanding of the combined capacity model of these services. Simply being able to back out what the bottleneck resource was during the period where eqiad was depooled would be a significant help (e.g., from a quick spot-check of thumbor worker container CPU, it's seemingly not that, so presumably some sort of concurrency ceiling somewhere) (T376516).

Edit: Looking back at T357547#9645651, an initial mitigation for the thumbor -> swift routing issue seen during the March switchover was to repool swift in codfw (and per the Day 8 sre.discovery.datacenter logs, it was left in that state). Thus, it's hard to tell whether a similar capacity issue might have eventually been observed, had it been left depooled (though noting that we seemingly did handle codfw being depooled for ~ 20h in that case).

CDN traffic in NA: As described in T370962#10178173, we're currently operating with eqiad pooled for edge / CDN traffic, but depooled for services (except opt-outs, and notably that includes swift).

In addition to testing our tooling / process, another goal of the switchover is to surface latent capacity issues should we lose a core site. Depooling the previously-primary site for CDN traffic for a period (typically 8 days) is part of that experiment, but is complicated this time around by the fact that ulsfo is also depooled due to T375345. Ultimately, we ended up repooling eqiad early after indications that we were stressing egress capacity in codfw.

We need to follow-up and assess whether our "design rules" for CDN / edge network (i.e., capacity and concurrent failures) hold in NA at the moment, and what follow-up actions might be necessary, if any. Stated differently, this is a question of what we can sustain both without and with additional intervention (e.g., some manner of manual traffic engineering on peering / transit ports, temporary changes to network monitoring, etc.). The switchover mainly emphasizes the former (no additional intervention), but can definitely be used to inform the latter.

Edit: After further discussion with Netops, it seems there was indeed slack capacity on some of the other peering or transit links out of codfw, which should have been sufficient to enable manual traffic engineering in a hypothetical scenario where eqiad could not be repooled. Together with other possible mitigations, I believe this answers the above question - i.e., we should be able to tolerate concurrent loss of both ulsfo and one of either eqiad or codfw, but this may require manual intervention depending on external demand and routing conditions (which is our design point, in a sense).

Steady-state sizing of mw-web and mw-api-ext: These services are currently provisioned to serve all traffic from a single DC, using the targets from T371273.

We should probably revert to our previous multi-DC sizing to free up k8s resources, and figure out what action items there are (if any) for validating that k8s cluster capacity allows for single-DC serving at steady state (in other words, if we dip below that line, we should probably know) (T376519).

Mentioned in SAL (#wikimedia-operations) [2024-09-30T14:51:23Z] <swfrench@cumin2002> conftool action : set/pooled=true; selector: dnsdisc=shellbox-video,name=codfw [reason: Pooling shellbox-video in codfw before (re)pooling eqiad on Wednesday - T370962]

Change #1076823 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] mw-web: increase replica counts by 15%

https://gerrit.wikimedia.org/r/1076823

Change #1075981 merged by Scott French:

[operations/puppet@production] scap: remove stale production dsh groups

https://gerrit.wikimedia.org/r/1075981

Change #1077111 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/cookbooks@master] sre.discovery.datacenter: exclude swift-https

https://gerrit.wikimedia.org/r/1077111

swfrench@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: Datacenter Switchover - T370962 started.

Mentioned in SAL (#wikimedia-operations) [2024-10-02T15:07:44Z] <swfrench@cumin1002> START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: Datacenter Switchover - T370962

swfrench@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: Datacenter Switchover - T370962 completed.

Mentioned in SAL (#wikimedia-operations) [2024-10-02T15:27:22Z] <swfrench@cumin1002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter Switchover - T370962

Alright, at this point, the September 2024 switchover is officially done. There are still a couple of follow-ups to wrangle (see T370962#10183874), which I'll work through / split off in the coming days.

Change #1075625 merged by jenkins-bot:

[operations/cookbooks@master] sre.discovery.datacenter: exclude kartotherian-ssl

https://gerrit.wikimedia.org/r/1075625

Change #1077111 merged by jenkins-bot:

[operations/cookbooks@master] sre.discovery.datacenter: exclude swift-https

https://gerrit.wikimedia.org/r/1077111

Change #1076823 abandoned by Scott French:

[operations/deployment-charts@master] mw-web: increase replica counts by 15%

Reason:

This was not needed

https://gerrit.wikimedia.org/r/1076823

All remaining follow-ups in T370962#10183874 have been split off to tasks or other discussion venues. Since nothing else is explicitly tracked here, I will resolve this.

Many thanks to all who contributed to making this a successful switchover!