Page MenuHomePhabricator

Consider rate limiting non-standard thumbnail sizes
Open, MediumPublic

Description

In T360589 MediaWiki started enforcing thumbnail steps $wgThumbnailSteps = [ 20, 40, 60, 120, 250, 330, 500, 960, 1280, 1920, 3840 ]; This increased the CDN hit-rate for thumbnail requests and opened the door for rate limiting requests targeting thumbnail sizes outside that list

Event Timeline

I asked for someone with global interface admin rights to change wikimini atlas size.

It looks like WikiMiniAtlas is maintained in a GitHub repo, I proposed a patch: https://github.com/dschwen/wikiminiatlas/pull/42 and I can copy it over to the wikis once it's accepted.

@dschwen Hi, perhaps I can reach you this way (I tried GitHub and on-wiki). I would appreciate if you could have a look at https://github.com/dschwen/wikiminiatlas/pull/42, so that we wouldn't have to diverge the on-wiki versions from the original one.

@matmarex I'm thankful for the work you have done on chasing down and getting it updated.

FWIW:

spark-sql (default)> select cache_status, split(split(uri_path, '/')[7], 'px-')[0] as thumbsize, count(*) as hitcount from wmf.webrequest where webrequest_source = 'upload' and year = 2025 and month = 9 and day = 11 and http_status = 200 and uri_path like '/wikipedia/%/thumb/%' group by split(split(uri_path, '/')[7], 'px-')[0], cache_status order by hitcount desc limit 500;
cache_status    thumbsize       hitcount
hit-front       250     445419228
hit-front       40      406752271
hit-front       60      377696058
hit-front       500     303876518
hit-front       120     289042157
hit-front       20      179218479
hit-front       330     117321836
hit-local       250     112543718
hit-local       500     77249711
hit-front       960     50656865
hit-local       120     38075975
hit-local       330     35981701
miss    250     35351020
hit-local       60      31749365
hit-front       70      29127271
hit-front       480     25139223
miss    120     22028672
miss    500     18750092
hit-local       960     18156635
hit-front       1200    15062849
miss    330     14256336
hit-local       480     11749335
hit-front       1280    10376642
hit-local       1280    7972151
hit-front       640     7639282
miss    960     7578566
hit-local       40      6614835
miss    60      6451716
hit-front       200     6445559
hit-front       15      6154554
hit-front       800     6069997
hit-front       100     5795980
miss    1280    5523177
hit-local       640     4730922
hit-front       865     4437661
hit-front       32      4427414
hit-front       150     4418979
miss    480     4203396
hit-front       1024    4105696
hit-front       80      4017913
miss    640     3879211
...

My rough estimate says it takes care of 81% of requests to upload.wikimedia.org. A lot of them are just hits to the original so they won't count but I couldn't exclude them.

By default MediaWiki core actually has a rate limit for this, but it seems we neglected to port this to Thumbor. It is based on wfThumbIsStandard (via wgThumbLimits, wgImageLimit, and wgResponsiveImages multiplication), and uses the renderfile-nonstandard bucket. This is separate from the recent introduction of wgThumbnailSteps.

We'll need a strategy here for InstantCommons. Right now ForeignAPIFile does not inherit any of of these features from the upstream repo, but from the local settings. So we'd need to make sure that at least the default values used by wfThumbIsStandard in MW 1.39 LTS+ are also permitted at the edge.

Today that is:

$wgThumbLimits = [ 120, 150, 180, 200, 250, 300 ];
$wgImageLimits = [
	[ 320, 240 ],
	[ 640, 480 ],
	[ 800, 600 ],
	[ 1024, 768 ],
	[ 1280, 1024 ],
	[ 2560, 2048 ],
];
$multipliers = [ 1, 1.5, 2 ];

widths = new Set;
for (m of $multipliers) {
  for (w of $wgThumbLimits) { widths.add(w*m); }
  for ([w] of $wgImageLimits) { widths.add(w*m); }
}
[...widths].sort((a,b) => a-b); 
//> Array(30) [120,150,180,200,225,240,250,270,300,320,360,375,400,450,480,500,600,640,800,960,1024,1200,1280,1536,1600,1920,2048,2560,3840,5120]

Note: This ignores the handling of tall images, which on File-description pages we link by height, and so produce an arbitrary width. wfThumbIsStandard currently accounts for this, and requires per-file metadata to resolve. That's easy in Thumbor, but impractical at the edge. The good news is that those links are much less used in practice, so could probably make due with a low but non-zero rate limit.

Long-term we may want to improve ForeignAPIFile so that it inherits wgThumbnailSteps config from the upstream repo and apply it locally (i.e. from Wikimedia Commons). That it it doesn't depend on local wgThumbnailSteps, which is currently disabled in MediaWiki core by default.

I don't know the details of ForeignAPIFile but the API endpoints actually return the correct standardized thumb urls. See for example: https://commons.wikimedia.org/w/api.php?action=query&titles=File:Albert%20Einstein%20Head.jpg&prop=imageinfo&iiurlwidth=100&iiprop=url and since this is on our side, it should be technically automatically deployed everywhere. Obviously, if InstantCommons/ForeignAPIFile uses another API endpoint, then it's a different story. I'd argue we still could rate limit and encourage people to either use standard sizes or build a cache. We are not fully disallowing it, just pushing for a more responsible use of the infra by rate limiting excessive requests.

I don't know the details of ForeignAPIFile but the API endpoints actually return the correct standardized thumb urls. See for example: https://commons.wikimedia.org/w/api.php?action=query&titles=File:Albert%20Einstein%20Head.jpg&prop=imageinfo&iiurlwidth=100&iiprop=url and since this is on our side, it should be technically automatically deployed everywhere. Obviously, if InstantCommons/ForeignAPIFile uses another API endpoint, then it's a different story. I'd argue we still could rate limit and encourage people to either use standard sizes or build a cache. We are not fully disallowing it, just pushing for a more responsible use of the infra by rate limiting excessive requests.

I enabled InstantCommons locally and tested it. I have had thumb steps and everything related disabled and it still got me 120px when I asked for 90px or 100px. So it really should work out of the box.

I don't know the details of ForeignAPIFile but the API endpoints actually return the correct standardized thumb urls. See for example: https://commons.wikimedia.org/w/api.php?action=query&titles=File:Albert%20Einstein%20Head.jpg&prop=imageinfo&iiurlwidth=100&iiprop=url […]

Right! I assumed the transformVia404 option triggers a fast path in ForeignAPIFile to format URLs without API call. This what we do in MediaWiki for LocalFile and ForeignDBFile, but ForeignAPIFile indeed still calls the API every time. That's not efficient, but I guess works in our favor on this one.

Likewise, Special:Redirect/file/Example.svg?width=12 also calls the API for InstantCommons consumers and thus honors steps, redirecting to https://upload.wikimedia.org/wikipedia/commons/thumb/8/84/Example.svg/20px-Example.svg.png.

In client-side code we do have mw.util.parseImageUrl and its resizeUrl() method. This doesn't honor steps and applies local config to foreign files. But, this isn't used by anything in MediaWiki core, and the only extensions calling it are essentially WMF-only (MediaSearch, Wikistories, ReaderExperiments) that need to be manually configured either way.

Nevermind!

Change #1228608 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[mediawiki/extensions/TimedMediaHandler@master] TimedMediaThumbnail: Set physical width and height

https://gerrit.wikimedia.org/r/1228608

Change #1230378 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[mediawiki/extensions/TimedMediaHandler@wmf/1.46.0-wmf.12] TimedMediaThumbnail: Set physical width and height

https://gerrit.wikimedia.org/r/1230378

Change #1230378 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@wmf/1.46.0-wmf.12] TimedMediaThumbnail: Set physical width and height

https://gerrit.wikimedia.org/r/1230378

Mentioned in SAL (#wikimedia-operations) [2026-01-22T16:27:38Z] <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1230378|TimedMediaThumbnail: Set physical width and height (T402792)]]

Mentioned in SAL (#wikimedia-operations) [2026-01-22T16:29:42Z] <ladsgroup@deploy2002> ladsgroup: Backport for [[gerrit:1230378|TimedMediaThumbnail: Set physical width and height (T402792)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Change #1228608 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] TimedMediaThumbnail: Set physical width and height

https://gerrit.wikimedia.org/r/1228608

Mentioned in SAL (#wikimedia-operations) [2026-01-22T16:36:42Z] <ladsgroup@deploy2002> Finished scap sync-world: Backport for [[gerrit:1230378|TimedMediaThumbnail: Set physical width and height (T402792)]] (duration: 09m 04s)

Mentioned in SAL (#wikimedia-operations) [2026-02-04T16:13:44Z] <Amir1> bumping rate limit of non-standard thumb sizes to medium browser score (T402792 T414805)

Mentioned in SAL (#wikimedia-operations) [2026-02-11T13:28:09Z] <Amir1> re-enabling rate limit of non-standard thumbnail sizes on medium browser score (T414805 T402792)