I'm not fond of the idea of not sending purges for indirect edits nor using RefreshLinksJob instead of HtmlCacheUpdateJob (too slow IMO).

I'm OK with:

1. Make the CDN cache purge aspect of jobs no-op if CURRENT_TIME > (rootJobTimestamp + CDN TTL + MAX NORMAL DB LAG); the page_touched part still happens
2. Lower $wgCdnMaxAge to 1 day; even if the CDN purge didn't run yet, the if-not-modified cache re-validations from CDN will save updated content if page_touched was updated
3. Deprioritizing and logging large high-use template backlink purge jobs a bit more

Skipping backlink CDN purges for templates/entitites/files with millions of backlinks might work, assuming the editor is logged in and not showing things to logged out users...I can't think of a clever way to maintain expectations.

In T250205#6154883, @aaron wrote:

I'm not fond of the idea of not sending purges for indirect edits

Agreed. The proposal to stop sending these purges does not stand by itself but rather would be an implementation step of "moving" the purges from one place to another (remove here, add there).

In T250205#6154883, @aaron wrote:

I'm not fond of […] using RefreshLinksJob instead of HtmlCacheUpdateJob (too slow IMO).

[…] Skipping backlink CDN purges for templates/entitites/files with millions of backlinks might work, assuming the editor is logged in and not showing things to logged out users...I can't think of a clever way to maintain expectations.

I think for someting like making a change to Template:Information or Template:Infobox, it's more valuable for logged-out users that pages get served quickly, than for those indirect changes to be applied instantly (e.g. with a cache miss resulting in a 2-60 second blank screen, awaiting an expensive reparse, possibly hitting the lower timeout threshold from GET compared to on-edit/job).

As I understand it, that last part is exactly what we're proposing. Although it would work as you'd like for unregistered editors as well, I think? They too get a full sessiont that bypasses the CDN and lasts for days/weeks (tied to their browser, incl typical session restore/continuation).

In a nut shell:

• Still purge from edit.
• Still bump page_touched recursively from a (quick) job. This means any natural cache miss or passthrough (editor with session) will still lazy re-parse as needed.
• Move recursive purges away from the quick job that does the page_touched bumps, and move it to the job that does the re-parses.

As additional optimisations we could:

• Make the job aware of the CDN TTL and forego the purge if that amount of time has passed since the root job started (currently 4 days I think).

Given that the reparse jobs are graceful (fixed size batches, each subsequent batch goes to the back of the queue), I don't think we need any special code for treating high-use templates differently. Naturally templates with only a few host pages would need only one or two batches and thus make it through the queue presumably within minutes/hours either way.

If we consider 4 days too long as the maximum time it could take for a recursive template update to apply, I'd rather we invest in making the reparse queue work faster (more resources and/or more efficient), rather than lowering the TTL in general and exposing more cache misses and on-demand reparses during a page view.

As a long-term direction, I'd actually be interested in exploring what MDN wiki does - to logically forbid re-parsing on GET as matter of principal (MDN only parses on POST or from jobs, their rev-latest parser cache as no TTL, and in the event stuff known-stale, the user is informed that it might be stale; while a high prio job works to fix it asap). Anyway, that's for a separate task :)

Change 598114 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Avoid HtmlCacheUpdateJob purges for content no longer in CDN

https://gerrit.wikimedia.org/r/598114

Change 598114 merged by jenkins-bot:
[mediawiki/core@master] Avoid HtmlCacheUpdateJob purges for content no longer in CDN

https://gerrit.wikimedia.org/r/598114

In T250205#6158994, @Krinkle wrote:

In T250205#6154883, @aaron wrote:

I'm not fond of the idea of not sending purges for indirect edits

Agreed. The proposal to stop sending these purges does not stand by itself but rather would be an implementation step of "moving" the purges from one place to another (remove here, add there).

In T250205#6154883, @aaron wrote:

I'm not fond of […] using RefreshLinksJob instead of HtmlCacheUpdateJob (too slow IMO).

[…] Skipping backlink CDN purges for templates/entitites/files with millions of backlinks might work, assuming the editor is logged in and not showing things to logged out users...I can't think of a clever way to maintain expectations.

I think for someting like making a change to Template:Information or Template:Infobox, it's more valuable for logged-out users that pages get served quickly, than for those indirect changes to be applied instantly (e.g. with a cache miss resulting in a 2-60 second blank screen, awaiting an expensive reparse, possibly hitting the lower timeout threshold from GET compared to on-edit/job).

As I understand it, that last part is exactly what we're proposing. Although it would work as you'd like for unregistered editors as well, I think? They too get a full sessiont that bypasses the CDN and lasts for days/weeks (tied to their browser, incl typical session restore/continuation).

In a nut shell:

• Still purge from edit.
• Still bump page_touched recursively from a (quick) job. This means any natural cache miss or passthrough (editor with session) will still lazy re-parse as needed.
• Move recursive purges away from the quick job that does the page_touched bumps, and move it to the job that does the re-parses.

That could work. The jobs that do the CDN purge would need to check (and possibly) update page_touched for sanity (in case the page_touched jobs did not already run).

To traffic and SRE folks: Where are we now on this with regards to load and priority? Was the reduction "enough" that the other ideas about cascading updates can wait until Q2?

Removing SRE, has already been triaged to a more specific SRE subteam(2 of them in fact)