This seems to be mostly rest base (FYI perf)

The schema https://schema.wikimedia.org/repositories//primary/jsonschema/resource_change/1.0.0.json allows to include the root event's timestamp causing the PURGE request. Considering that the CDN has a max cap on TTL set at 24h we could safely exclude PURGEs generated by events older than this TTL. Sadly mediawiki and rest base don't currently set root event information:

{"$schema":"/resource_change/1.0.0","meta":{"uri":"http://en.wikipedia.org/api/rest_v1/page/mobile-sections-remaining/File_talk%3ABookworm_Adventures.jpg/414315846","stream":"resource-purge","domain":"en.wikipedia.org","dt":"2022-09-16T08:44:30.651Z","id":"d00a1cae-359b-11ed-b732-295dce798348","request_id":"d00a1caf-359b-11ed-b732-295dce798348"}}
{"$schema":"/resource_change/1.0.0","meta":{"uri":"https://de.m.wikipedia.org/wiki/Vegemite","request_id":"1ffc9449-0501-458d-b39c-12ae655bcd08","id":"93b223d4-270d-454c-bdf9-0f4265b419ee","dt":"2022-09-16T05:49:02Z","domain":"de.wikipedia.org","stream":"resource-purge"},"tags":["mediawiki"]}

non-PURGE requests VS PURGE requests hitting ats@cp3050 during the last 30 days:

Which dashboard is this from? (There's quite a few in Grafana for "frontend", "traffic", "ATS" or "Varnish").

I was able to find one a bit like the above for one specific server, Grafana: ATS Instance Drilldown, cp3050.

This appears fairly stable over the past three months. Has something changed that makes this a specific issue today? What threshold should we aim for? What would the cost be of maintaining current levels as-is? (e.g. infrastructure improvements).

Past issues:

• T249325: cache_text cluster consistently backlogged on purge requests
• T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
• T133821: Make CDN purges reliable

General open issue:

• T250205: Reduce rate of purges emitted by MediaWiki

The dashboard is https://grafana.wikimedia.org/d/kHk7W6OZz/ats-cluster-view?orgId=1&var-site=esams&var-layer=backend&var-cluster=text&from=now-30d&to=now.
We also have a purged specific dashboard showing data per instance: https://grafana.wikimedia.org/d/RvscY1CZk/purged?orgId=1.

We are aware that this isn't a new issue @Krinkle, this task is the consequence of @Ladsgroup and myself talking during the SRE Summit in Prague. Happy to tag this as a duplicate of T250205 if required, but it seems that not only MW is responsible for a big amount of PURGEs but rest base as well

@Krinkle Note that it's not mediawiki, 88% of the purges are something like domain.org/api/rest_v1/ and coming from restbase.