A few explorations to get started:

Reduce non-pageview workload

Reason for T149873: CentralNotice: Allow using Varnish cache of Special:BannerLoader HTML for logged-in users:

In T149873#12134136, @Krinkle wrote:

[…]
It is desirable to get shared caching going on here, because we know the response is a fairly straight forward key-value retrieval that doesn't need to vary by anything other than its URL.

It should be feasible as well, because we already do the same thing in other entrypoints (such as load.php) and last I checked there was nothing in Varnish to make that work. It's up to MediaWiki to decide whether to emit Cache-Control:public and to not emit Vary:Cookie. […] [This] work will benefit […] in two ways:

1. We can apply what we learn to other entrypoint where we'd like to consider shared CDN caching for more logged-in user responses (as we already do for ResourceLoader load.php).
2. By CDN-caching CentralNotice banner HTML, we free up MediaWiki backend load for logged-in pageviews.

Making MediaWiki serve pageviews faster

I suggest we explore these ones:

• T347123, T428517: Reduce database time by avoiding needless queries. Baseline: ~100ms.
• T318306: Reduce Memcached time by using a faster serializer. Baseline: ~80ms.
• T270525: Reduce Mustache time by using a faster engine. Baseline: ~20ms.
• T225074: Reduce Apcu time by using a faster serializer. Baseline: ~20ms.
• T374761: Revert overhead from PageAssessment extension. Baseline: ~20ms.
• T432518: Revert overhead from HCaptcha. Baseline: ~20ms.
• T432882: Reevaluate "PageNotice" approach to avoid overhead on most articles. Baseline: ~40ms.
• T432883: Revert Disambiguator overhead from interface messages. Baseline: ~5ms.
• TODO: Reduce PageTriage Hooks::onArticleViewFooter overhead. Baseline: ~35ms.

This is based on analyzing a pageview (logged-out, cache miss via WikimediaDebug, mw-experimental-eqiad with 1 warmup for opcache and ParserCache, instrumenting a hard refresh): https://performance.wikimedia.org/excimer/profile/b630781ba01d857c.

The total pre-send latency is 470ms (/w/index.php, sans postOutputShutdown), and that's logged-out, with a ParserCache hit, on a warmed-up server. This used to be ~250ms not too long ago.

Bottom-up:

• [@100ms] ~100ms: MySQL connections and queries (Rdbms DBConnRef, Rdbms Database)
• [@180ms] ~80ms: Memcached gets (MemcBagOStuff, WANObjectCache)
• [@200ms] ~20ms: Apcu cache fetches (ApcuBagOStuff)
• [@240ms] ~40ms: ParserCache::restoreFromJson (50-50 between FormatJson::parse and JsonCodec::newFrom)
• [@260ms] ~20ms: Mustache formatting (TemplateParser::processTemplate, eval, LightnCandy)
• [@280ms] ~20ms: IcuCollation to compute sort keys for category names in OutputPage
• [@300ms] ~20ms: Autoloader (i.e. load thousands of classes from opcache, in theory no disk access or compiling involved). This varies from request to request depending on how much code it ends up loading. For example this can be ~9ms on load.php requests.

Top-down based on 250ms-400ms for MediaWiki Backend Pageview Timing. I've marked regressions and other surprises in bold.

• [@15%] 15% ~60ms : ParserOutputAccess to obtain ParserOutput
  • ~20ms: MemcBagOstuff fetch via ParserCache. I expected this to be ~1ms. Something is wrong here.
  • ~40ms: Unserialize and decode the object via ParserCache::restoreFromJson. Even with memcached being 20X slower than it should be, we spending another 20ms on parsing the JSON (json_decode), and then another 20ms constructing a PHP class (JsonCodec). This is almost certainly a regression compared to the PHP unserialize call we just to have (ref T161647).
• [@25%] 10% ~40ms : Article::showNamespaceHeader uncached wikitext parsing and Scribunto calls. This was enabled on article pages last year in T151682 as a way to achieve "PageNotice" functionality (before that, this function was limited to talk pages).
• [@30%] 5% ~20ms: SkinMustache formatting HTML (TemplateParser::processTemplate, LightnCandy)
• [@35%] 5% ~20ms : PageAssessments extension Hooks::onParserAfterParse adds overhead to Message->parse calls for interface messages, mistaking them regular page parses. It performs uncached database queries during 14 separate interface message parses. From a quick git-blame, it appears to have been introduced last month for T374761.
• [@40%] 5% ~20ms : ConfirmEdit (HCaptcha) extension Hooks::onMakeGlobalVariablesScript is calling User::getBlock (via addIPBlocksScoreCollectionVars) on every pageview. This is a regression introduced last month in T426943. It adds 5% to all latencies through multiple network roundtrips (new connection to GlobalBlocking DB, DB queries, memcached queries). At glance, this doesn't actually work because logged-out pageviews are CDN-cached so this is either an expensive no-op or actively poisoning caches. I've confirmed just now that when perfoming a logged-out pageview from a VPN connection (one that we block from editing) my cache miss poisoins the CDN cache with HCaptchaBlockedIpEditingScoreCollectionSiteKey.
• [50%] 10% ~35ms : PageTriage extension Hooks::onArticleViewFooter is obtaining a second ParserOutput object, separate from and without re-using the main one. I considered whether the cost may be misattributed here (if this one runs first) but that's not the case, the chronological flame graph shows clearly two separate calls and this one is after the main one. (Image below). This means we've got a second network roundtrip to Memcached and the whole JSON unserialize. Fixing this to reuse the main ParserOutput should eliminate this cost entirely, and cut the ParserOutputAccess call above in half. It expains at least part of why that one was so high, because it's actually two calls, not one. I've confirmed via an XHGui profile that were indeed 2 calls (not more).
• [52%] 2% ~5ms : Disambiguator ext Hooks::onGetLinkColours, similar to the PageAssessments issue, but less severe because only 2 of the wikitext-parsed interface messages on this page contain links. It is adding overhead by querying the database separately for each linked page title to add the mw-disambig class. They are however uncached database queries that multiply by N for each parsed interface message on the page that contains a link. It was introduced in 2014 (T10339). During a normal page parse this is batched, but because each message is parsed separately, this is ineffective. I suggest we limit this to page content by turning it off for interface messages.

We can also observe these percentages via the aggregate MediaWiki Entrypoint Profiling: One component dashboard. The percentages are lower here since they take into account all action=view traffic, including special pages, diffs, and HTTP 304 responses:

ext_PageAssessments (30d)
ext_ConfirmEdit (90d)
ext_PageTriage (6M)
ext_Disambiguator (6M)
core_Json (1y)
lib_Rdbms (6M)
lib_ObjectCache (6M)

We booked a nice win this first week. Two fixes rolled with the train this week, both reversing a 5% latency regression (PageAssessments T374761#12134375 and HCaptcha T432518).

From the Grafana: Backend Pageview Timing dashboard:

• The p75 quantile is about 10% or 20ms lower than the same time last week (240ms → 220ms).
• About 5% of traffic moved from the >250ms bucket to <=250ms.
• And another 5% moved from >100ms to <=100ms.

I guess a lot of requests were hovering near the 100ms and 250ms thresholds.

Change #1323826 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] Setup: Avoid SessionManager::shutdown overhead on load.php and CLI

https://gerrit.wikimedia.org/r/1323826

Change #1323826 merged by jenkins-bot:

[mediawiki/core@master] Setup: Avoid SessionManager::shutdown overhead on load.php and CLI

https://gerrit.wikimedia.org/r/1323826

Progress update:

• Done T297300: PageProps doesn't cache result when it is false
• Done T426246: Improve imageinfo API to return a set of thumbnail URLs
• Done T77391: MediaViewer should not make an API request for each reuse size
• Done: Skip unneeded SessionManager shutdown on logged-out index.php and api.php requests https://gerrit.wikimedia.org/r/1323826
• In progress: T432883: Avoid DB lookups from Disambiguator when rendering interface messages
• In progress: T432882: Avoid significant "subjectpageheader" parsing on every pageview
• In progress: T428517: The user's row is being loaded three times on every page view
• In progress: T56037: ForeignAPIRepo / InstantCommons should fetch multiple-density thumbnails in one request

The audit at T432882#12209205 reminded me of the various ways wikis handle display of padlock icons on protected pages, so I analysed the state of this and wrote it up at https://nl.wikipedia.org/wiki/MediaWiki:Gadget-ProtectionTemplates.js:

There are broadly two ways protection icons are managed on MediaWiki wikis:

@Krinkle wrote at nl:MediaWiki:Gadget-ProtectionTemplates.js

1. The wiki has a magic {{pp}} template that checks {{PAGENAME}} or {{PROTECTIONLEVEL}} to determine what and when to add an icon. The icon is rendered via <indicator>. This is what English Wikipedia does, see https://en.wikipedia.org/wiki/Template:Pp.
   • Downside: Requires a complex template.
   • Downside: Requires a bot, or manual edits by admins when they enable protection. Note that it is okay if {{pp}} is left afterward protection is changed/removed/expired, because it automatically renders the right thing or nothing on unprotected pages.
   • Upside: It is easy to understand and find where the icon comes from.
   • Upside: No JavaScript.

2. The wiki a gadget (this gadget) to automatically display the icon. This is what Dutch Wikipedia does.
   • Downside: Requires that each template is static. These templates MUST NOT use {{PAGENAME}} or {{PROTECTIONLEVEL}} to change what and when to output something.
   • Downside: Requires JavaScript
   • Upside: No manual edits.
   • Upside: No bots.

I then optimised the gadget (nlwiki revision 71765969) to make its Parse API requests CDN-cachable and browser-cachable such that it effectively becomes a bundled set of HTML snippets that the gadget chooses between. (That might actually be a sensible next step.)

Impact (Turnilo query):

• Before: ~90K/day requests to api.php?action=parse, of which 0% are CDN cache hits, and 100% CDN passes.
• After: ~80K/day requests to api.php?action=parse, of which ~60% are CDN cache hits, ~1% are CDN cache misses, and 39% are CDN passes.

I exclude requests rejected by the CDN (Cache status: int-tls or int-front, i.e. HTTP 429) as these don't reach MediaWiki and don't cost us much. I also excluded requests without a Mozilla/5.0 user agent as a crude way to focus on browser traffic (because user_agent_map isn't in this dataset, and X-Is-Browser is broken per T435152).

In T432488#12224542, @Krinkle wrote:

[…] I analysed the state of this and wrote it up at https://nl.wikipedia.org/wiki/MediaWiki:Gadget-ProtectionTemplates.js: […]

I then optimised the gadget (nlwiki revision 71765969) to make its Parse API requests CDN-cachable and browser-cachable such that it effectively becomes a bundled set of HTML snippets that the gadget chooses between. (That might actually be a sensible next step.) […]

Done. Now that Aug 19 has ended in UTC, we have a full day's worth of data.

• Before: ~90K/day requests to api.php?action=parse (0% CDN cache hit, 100% CDN pass).
• After (first change): ~80K/day requests to api.php?action=parse, of which ~60% are CDN cache hits (and ~1% CDN cache miss, 39% CDN pass).
• After (second change): ~40/day requests to api.php?action=parse (0% CDN cache hit, 100% CDN pass).

https://w.wiki/TcBr

Change #1340044 had a related patch set uploaded (by Krinkle; author: SomeRandomDeveloper):

[mediawiki/core@master] Skin: Avoid unnecessary AuthManager checks if they're not needed

https://gerrit.wikimedia.org/r/1340044