Traffic team has moved 14 hosts from upload to text which should have increased its capacity and thus reducing the load (T436363). I don't know if we should move more or problem lies somewhere else.

Traffic could you help us looking into this ?

Change #1344212 had a related patch set uploaded (by Blake; author: Blake):

[operations/deployment-charts@master] mw-web: upsize for single-DC serving.

https://gerrit.wikimedia.org/r/1344212

As an action to at least partially unblock the switchover, we could finally move to block all requests from clients that:

• have a very low browser score
• do not honor our UA policy

it is a very radical action, but it would shave off about 15-20% of all uncached requests to mw-web: see https://w.wiki/V5tD

In T438896#12353980, @Joe wrote:

As an action to at least partially unblock the switchover, we could finally move to block all requests from clients that:

• have a very low browser score
• do not honor our UA policy

it is a very radical action, but it would shave off about 15-20% of all uncached requests to mw-web: see https://w.wiki/V5tD

I would say this is overdue and we should enable that rule.

Change #1344212 merged by jenkins-bot:

[operations/deployment-charts@master] mw-web: upsize for single-DC serving.

https://gerrit.wikimedia.org/r/1344212

Mentioned in SAL (#wikimedia-operations) [2026-09-23T10:22:07Z] <blake@deploy1003> Started scap sync-world: Upsize mw-web T438896

Mentioned in SAL (#wikimedia-operations) [2026-09-23T10:25:40Z] <blake@deploy1003> Finished scap sync-world: Upsize mw-web T438896 (duration: 04m 20s)

Mentioned in SAL (#wikimedia-operations) [2026-09-23T10:38:17Z] <jayme> sudo confctl --quiet --object-type discovery select 'dnsdisc=mw-web-ro' set/ttl=10 - T438896

<effie> eqiad has double the memcached capacity codfw has atm, as we are waiting for the refresh. While not the sole reason for the swithover issues, it is a contributing factor
<effie> iirc (please check), codfw still has extstore enabled, using disk to cache objects-to-be-evicted, this could be increased easily, though it will require restarting the daemons

T352885: Enable extstore to a subset of memcached servers (experiment)

Discussing it on IRC some this morning, we decided that it was also worth trying a small deviation from the normal procedure, and leaving eqiad's CDN edge pooled. The idea is that the working set of hot content differs between eqiad and codfw, so, let's keep our eqiad cache capacity online, and not force a larger working set into codfw's edge.

This isn't that unfaithful to an "eqiad deleted" scenario -- in such a case we would probably consider redistributing eqiad's load across a few other sites anyway.

If this works out, it would still allows us to test mw-web capacity, do eqiad database maintenances, etc.

eqiad has been pooled for a little over half an hour, and the "true hitrate" (hit/(hit+miss)) is at 77% and climbing:

While eqiad was depooled, codfw had this figure around 81-82%, so eqiad is looking pretty warm already.

In T438896#12353980, @Joe wrote:

it is a very radical action, but it would shave off about 15-20% of all uncached requests to mw-web: see https://w.wiki/V5tD

This rule was activated during testing today and it immediately led to noc feedback and CI pipelines in gitlab failing so it was disabled again.

Things that where found in mw traces:

• Long POSTs to eventgate-main, timing out after 60s. We should probably set a short, hard timeout for these? (https://trace.wikimedia.org/trace/3d7600eb886c5a7626d88431930f7aaa)
• Huge number of consecutive SELECTs from Campaign::getCampaignMixins (https://trace.wikimedia.org/trace/39a8652b5280758a84f504a511d87e38?uiFind=bce3754db950a359)

In T438896#12356442, @JMeybohm wrote:

Things that where found in mw traces:

• Long POSTs to eventgate-main, timing out after 60s. We should probably set a short, hard timeout for these? (https://trace.wikimedia.org/trace/3d7600eb886c5a7626d88431930f7aaa)

This has been an issue in previous outages (2025-05-21 when we discovered the cpu governor issue, notably) where these timeouts tying up workers lead to amplified pressure elsewhere in the infra. We should definitely drop this to something low

In T438896#12356442, @JMeybohm wrote:

• Huge number of consecutive SELECTs from Campaign::getCampaignMixins (https://trace.wikimedia.org/trace/39a8652b5280758a84f504a511d87e38?uiFind=bce3754db950a359)

@VPuffetMichel Hi, would you mind asking someone in the Connection team to take a look at this? Thank you!

Change #1344739 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] services_proxy: drop eventgate timeout to 11s

https://gerrit.wikimedia.org/r/1344739

Change #1344739 merged by Hnowlan:

[operations/puppet@production] services_proxy: drop eventgate timeout to 11s

https://gerrit.wikimedia.org/r/1344739

So before the big ol' power outage made everyone's day a lot more eventful, we had been trying to do the switchover to codfw again.

Our first attempt to move eqiad's read-only traffic to codfw failed -- appservers saturated and latencies skyrocketed. So we increased the number of appserver pods, hoping that was the bottleneck, and tried again, for about 10 minutes from 15:04 UTC to 15:15 UTC. Same thing happened even after more appserver capacity, suggesting the bottleneck was in fact elsewhere, and so we went looking for where and why (until the power outage struck).

I downloaded all the mw-web @ codfw traces from this particular overload window, 2026-09-23 15:04-15:15 UTC, and I vibe-scripted-analyzed them together with Claude. We only have traces for 0.1% of the traffic from interval, but for those 769 requests, we have extremely detailed timing information for different stages of Mediawiki's processing. I think they leave us with some very good pointers on what went wrong.

Big takeaways

• The single biggest contributor to the overload was the cumulative queuing times in Envoy + Apache: 1138.9 seconds (!!) across 769 requests. (This time also includes PHP worker startup, but that should be small.)

• In the initial minutes of the outage, we spent a lot of Mediawiki time on s7 database calls, which caused many queued requests to pile up -- mostly on load.php. Once the various caches had been somewhat warmed, this subsided. But after that, Mediawiki re-focused its death ray on eventgate and/or kafka, and in the later minutes we spent much time blocked waiting for eventgate-main to flush in post-send.
  • We saw a similar but much worse version of this during the eqiad capacity shortfall while codfw was lights-out, where it appears that almost all Mediawikis were blocked on eventgate/kafka for perhaps as long as 10 minutes (!!)

• After queuing time, the largest single cost in MW was DB time: 87.1s of cumulative database wall time, predominantly against s7 replicas.
  • 35% of this DB time (30.1s/87.1s) comes from the BEGIN/USE/SET/COMMIT dance that Mediawiki RDBMS does upon every new database connection. Again, mostly from s7 replicas, which must have been under very high contention to be responding so slowly to almost-no-op queries.
  • Another 30% (26.0s) of the database time comes from the single traced request that executes the Campaign::getCampaignMixins queries. It shows as 44 SELECTs made in quick succession to one s7 replica, taking on average 600ms each.
• Close behind: 83.2s cumulative time blocked waiting on eventgate-main (this very likely under-counts the true blocked time)
• 51.1s cumulative time on getParserOutput and children (n.b. some database time is included in this)
• 45.9s cumulative WANObjectCache::getWithSetCallback wall time -- and as above some database time is included in this, including and especially 100% of the getCampaignMixins time.
  • There is no stampede protection on this CentralNotice code path that calls getCampaignMixins, and, the cache key is different for each (wiki, user interface language) pair. The cache keys in question are looked up by every load.php fetch that involves any sort of JS, even if ext.centralNotice.choiceData isn't requested, as in this trace.
  • The regeneration costs for this WANObjectCache key (see F105294207) also look to correlate with the September 1st suspected Mediawiki performance regression T438962.
  • Given that the regeneration makes N serial database roundtrips, where N is the number of relevant campaigns, and that, as best I can tell, the number of active campaigns greatly increased starting around 2026-08-31, this confluence of code and data is likely a huge source of the cold-start problem in codfw, and a significant contributor to overall Mediawiki appserver load (the bare minimum load is hourly recomputations of ~thousands of these keys).
  • Cold starting codfw is extra-hard because the traffic mix for wikis and user languages likely differs substantially between core datacenters.

What needs to change ASAP

MediaWiki-extensions-CentralNotice

• ChoiceDataProvider::fetchChoices must not call Campaign::getCampaignMixins for each campaign by name in a for loop; it should instead fetch mixins by campaign ID, all at once, with not_id IN (...), as it does for regions and countries just before the foreach.
• ChoiceDataProvider::getChoices must modify its WANObjectCache::getWithSetCallback() invocation to include stampede protection. I think all that is needed is adding something like 'lockTSE' => 30, to the options dictionary. (The long-extant comment is dangerously wrong under cold-start conditions.)

Event-Platform

• We must either always set ?hasty=true when calling EventGate from Mediawiki, even when in post-send, or, we must implement a much shorter (and perhaps randomized) timeout for EventGate requests.

What should probably change in the near term

DBA

• We have only 7 replicas of s7 in codfw, compared to 13 in eqiad. This very likely made the chained selects and lack of stampede protection much more harmful there. Is it worth upsizing?

ServiceOps

• Up-size the memcached cluster in codfw.

To consider in the longer term

• Some mechanism for proportional (weighted / partial) pooling of a DNS/Discovery service, even if the mechanism is "ATS-only" and/or gross. It's very likely we'll need this for cold starts for other disaster recovery scenarios, and it makes cold starts much safer.

Thank you for the investigation Chris! <3 I updated the title to reflect the fact that thumb migration is likely not the cause. Also the Campaigns is CN feature and I got confused. Sorry for the wrong ping Val.

xref T439239: CentralNotice must batch campaign mixin lookup & use stampede protection

In T438896#12371812, @Ladsgroup wrote:

Thank you for the investigation Chris! <3 I updated the title to reflect the fact that thumb migration is likely not the cause. Also the Campaigns is CN feature and I got confused. Sorry for the wrong ping Val.

+1 Thanks @CDanis !

& Thanks for updating the title @Ladsgroup, apologies for the false positive! Updating the description to reflect this similarly.

I put together a cool interactive data visualization for the overload event, along with some explainers. If you're interested in this incident, or the approach used to diagnose the causes, check it out: https://people.wikimedia.org/~cdanis/2026-switchover-overload/