Change 838872 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] objectcache: use WRITE_BACKGROUND in WANObjectCache where applicable

https://gerrit.wikimedia.org/r/838872

Change 838872 merged by jenkins-bot:

[mediawiki/core@master] objectcache: use WRITE_BACKGROUND in WANObjectCache where applicable

https://gerrit.wikimedia.org/r/838872

In T302623#8384645, @gerritbot wrote:

Change 838872 merged by jenkins-bot:

[mediawiki/core@master] objectcache: use WRITE_BACKGROUND in WANObjectCache where applicable

https://gerrit.wikimedia.org/r/838872

Impact on https://grafana.wikimedia.org/d/lqE4lcGWz/wanobjectcache-key-group, as measured by the most trafficked key group (SqlBlobStore_blob):

The total miss-compute time is all time spent in getWithSet during a cache miss. The regen_set_delay is the time up until right before the SET command (and possibly the cool-off bounce). This time has been cut by 0.5ms from 0.75ms to 0.25ms.

Would T124371: Clean up usage of $_SESSION in WMF-deployed extensions be worth looking into? I think PHPSessionHandler triggers lots of unnecessary session logic by registering SessionManager as a PHP session handler, which is only necessary for compatibility with old code which does normal PHP session operations, which is hopefully all gone by now, so we should be able to switch off $wgPHPSessionHandling in Wikimedia production, and maybe as a MediaWiki default.

In T302623#8566372, @Tgr wrote:

Would T124371 be worth looking into? I think PHPSessionHandler triggers lots of unnecessary session logic […]

If I understand correctly, wgPHPSessionHandling only controls compat code for $_SESSION. Turning it off is not the same as enabling MW_NO_SESSION, correct? (Although enabling MW_NO_SESSION does -also- disable wgPHPSessionHandling of course.)

I'd agree. We could as a first step set it to "warn" in beta and on test wikis. Then once WMF has reached the "disable" stage on all wikis, we can e.g. set it to "warn" for one MW LTS cycle.

In T302623#8599631, @Krinkle wrote:

If I understand correctly, wgPHPSessionHandling only controls compat code for $_SESSION. Turning it off is not the same as enabling MW_NO_SESSION, correct?

Correct. When it is $wgPHPSessionHandling is enabled, SessionManager is registered as the PHP session handler. In theory, this is entirely unnecessary for SessionManager to work; it's only needed to avoid a split-brain situation when some code invokes PHP session handling directly (by touching $_SESSION, calling session_start() etc). I'm not at all sure but I think it results in a bunch of extra calls to SessionManager even if nothing directly invokes PHP session handling, just because the interface for PHP session handlers is limited and awkward.

Change 890872 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] linker: Remove DB lookups from makeKnownLink() and improve docs

https://gerrit.wikimedia.org/r/890872

Change 902376 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@wmf/1.41.0-wmf.1] objectcache: Disable cool-off bounce feature

https://gerrit.wikimedia.org/r/902376

Change 830706 merged by jenkins-bot:

[mediawiki/core@master] objectcache: Remove WANObjectCache's internal cool-off bounce feature

https://gerrit.wikimedia.org/r/830706

Looks like we have a major regression in latency, starting in November.

Grafana: Backend Pageview Timing dashboard:

Going by DC is this isolated on desktop in a given DC? That's maybe an obvious and expected thing based upon the most active (or inactive) DC. I suspect the stuff around the 30th of December is related to some natural (or automated) spike, but more looking at the general trends here.

Any way to drill down on the wiki DB to see if it's confined?

In T302623#11499381, @Krinkle wrote:

Looks like we have a major regression in latency, starting in November.

Grafana: Backend Pageview Timing dashboard:

Grafana: Appserver RED dashboard (mw-web, any DC, GET, HTTP 200):

There does not appear to be a significant change in the overall rate or distribution of requests during this time (e.g. by DC, or HTTP method):

Looking back over the past 1.5 year, via Grafana: Backend Pageview Timing dashboard.

1 Jan 2025 - 19 Mar 2026 (in two pieces, because Prometheus enforces a 375 day maximum date range, regardless of interval, despite storing data for 5 years; ref T420676)

Detail Dec 2025 and Mar 2026

Annotations (p75 MediaWiki Backend Pageview Timing, not to confuse with overall mw-web webserver latency)

1. Jan 2025 Baseline: 235ms mobile / 230ms desktop.
2. Mar 2025 Regression (T394059): 300ms mobile +30% / 240ms desktop.
3. May 2025 Recovery (T394059): 280ms mobile / 230ms desktop. (Recovery appears incomplete due to other uninvestigated regressions).
4. Sep 2025 Baseline: 330ms mobile / 230ms desktop.
5. Nov 2025 Regression (T416540): 420-880ms mobile +30 to +160% / 240-480ms desktop (varies daily, numbers are from 12:00-23:00 which is the lowest point of the day and 02:00 AM which when we're halfway down from the midnight peak)
6. Mar 2026 Recovery: 415-660ms mobile / 245-310ms desktop (idem, 17:00 and 02:00)

Change #890872 abandoned by Hashar:

[mediawiki/core@master] linker: Remove DB lookups from makeKnownLink() and improve docs

https://gerrit.wikimedia.org/r/890872

Change #890872 restored by Krinkle:

[mediawiki/core@master] linker: Remove DB lookups from makeKnownLink() and improve docs

https://gerrit.wikimedia.org/r/890872