Measure request frequency of thumbnail sizes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	MatthewVernon
	Nov 17 2025, 5:58 PM

Description

In T408715 I compiled a list of canonical thumbnail sizes in use. The next question is: how frequently are these sizes used?

Using a convenience sample of a 24-hour period (2025-10-24) and the wmf.webrequest spark table (i.e. requests reaching the CDN):

the 8 wgThumbnailSteps account for 86% of thumbnail requests (and are the 8 most-requested sizes)
the 30 canonical sizes listed in T408715 account for 92% of thumbnail requests
the 30 most-requested sizes account for 94% of thumbnail requests
the 4 pregenerated thumbnail sizes account for 2% of thumbnail requests
the commonest non-canonical size is 480 (ranked 9), accounting for 1.4% of requests (434/second)
Over 8000 different thumbnail sizes were requested, from 1 to 121200
316 different thumbnail sizes were requested at least once per second (i.e. > 86400 per day)
Extracting thumbnail size from uri_path is not entirely straightforward

Taking our commonest 30 sizes, in rank order:

size	count	rps	%	cumulative %	canonical
250	555502137	6429	20.2	20.2	Y
60	397775830	4604	14.4	34.6	Y
500	374795875	4338	13.6	48.2	Y
40	362872476	4120	13.2	61.4	Y
120	303617631	3514	11.0	72.4	Y
330	158260177	1832	5.7	78.1	Y
20	154562318	1789	5.6	83.8	Y
960	71684322	830	2.6	86.4	Y
480	37502885	434	1.4	87.8	NO
1200	24267826	281	0.9	88.7	NO
70	22379621	259	0.8	89.4	Y
1280	21968978	254	0.8	90.2	Y
640	15191911	176	0.6	90.8	Y
800	14293204	165	0.5	91.3	Y
1024	9268869	107	0.3	91.6	Y
600	6392998	74	0.2	91.9	NO
200	6321587	73	0.2	92.1	Y
160	6259469	72	0.2	92.3	Y
1920	5913263	68	0.2	92.5	Y
320	5879692	68	0.2	92.7	Y
150	5567819	64	0.2	92.9	Y
2560	5556343	64	0.2	93.1	Y
512	5450688	63	0.2	93.3	NO
300	5441267	63	0.2	93.5	Y
375	5184308	60	0.2	93.7	NO
32	4838561	56	0.2	93.9	NO
80	3979487	46	0.1	94.0	Y
24	3831704	44	0.1	94.2	NO
100	3805802	44	0.1	94.3	NO
400	3600631	42	0.1	94.5	Y

After these top 30, we've accounted for 2601967679 / 2754743467 requests, leaving 5.5% requests amongst the remaining sizes (or a combined 1768 requests/second).

Taking our list of 30 canonical sizes in size order:

size	count	%	rank
20	154562318	5.6	7
40	362872476	13.2	4
60	397775830	14.4	2
70	22379621	0.8	11
80	3979487	0.1	27
120	303617631	11.0	5
150	5567819	0.2	21
160	6259469	0.2	18
180	3094778	0.1	33
200	6321587	0.2	17
220	2982701	0.1	34
240	2963146	0.1	35
250	555502137	20.2	1
260	941860	0.03	71
300	5441267	0.2	24
320	5879692	0.2	20
330	158260177	5.7	6
360	1973411	0.07	47
400	3600631	0.1	30
450	750619	0.03	79
500	374795875	13.6	3
640	15191911	0.6	13
800	14293204	0.5	14
960	71684322	2.6	8
1024	9268869	0.3	15
1280	21968978	0.8	12
1920	5913263	0.2	19
2560	5556343	0.2	22
2880	685425	0.02	83
3840	3695	<0.01	1285

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	constants: Force 480px to 500px	mediawiki/extensions/Popups	master	+4 -3

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		MatthewVernon	T408062 FY 25/26 WE 5.4.7 Standardize thumbnail sizes
		Resolved		MatthewVernon	T410304 Measure request frequency of thumbnail sizes

Event Timeline

MatthewVernon created this task.Nov 17 2025, 5:58 PM

MatthewVernon updated the task description. (Show Details)

MatthewVernon updated the task description. (Show Details)Nov 18 2025, 8:20 AM

A couple of notes on extracting thumbnail size from uri_path - a previous approach used

SELECT split(split(uri_path, '/')[7], 'px-')[0] as thumbsize

but this has a number of shortcomings, particularly that the array index of 7 is fragile, and incorrect for e.g. /archive/ thumbs. So I refined it somewhat to take the final path element, and then split that at px- and then split the result on - and take the final element (thus coping with prefix-NNNpx like you get with translated SVG files):

P84991 cope with both variable length uri_path (might or might not have /archive/ in) and that size may be NNNpx or prefix-NNNpx

1	select slice(split(split(slice(split(uri_path, '/'),-1,1)[0], 'px-')[0],'-'),-1,1)[0] as thumbsize, count(*) as hits from wmf.webrequest where webrequest_source = 'upload' and year = 2025 and month = 10 and day = 24 and hour = 10 and http_status = '200' and uri_path like '/wikipedia/%/thumb/%' group by thumbsize order by hits desc limit 10;

This still left a very few stragglers (15, mostly SVG files with URL-encodings in their names), which is likely good enough, but we can do better with a simple regexp:

select regexp_extract( slice(split(uri_path, '/'),-1,1)[0], '([0-9]+)px') as thumbsize, count(*) as hits from wmf.webrequest where webrequest_source = 'upload' and year = 2025 and month = 10 and day = 24 and http_status = '200' and uri_path like '/wikipedia/%/thumb/%' group by thumbsize order by hits desc;

This produces the same answers (modulo the 15 errors), is clearer, and only takes ~10% longer to run. Finally, of course, we can just do the whole operation with a single regexp - to match for thumbsize as previous and then state that it must be followed by only not-\ characters:

select regexp_extract(uri_path, '([0-9]+)px[^/]+$') as thumbsize, count(*) as hits from wmf.webrequest where webrequest_source = 'upload' and year = 2025 and month = 10 and day = 24 and http_status = '200' and uri_path like '/wikipedia/%/thumb/%' group by thumbsize order by hits desc;

MatthewVernon mentioned this in P84991 cope with both variable length uri_path (might or might not have /archive/ in) and that size may be NNNpx or prefix-NNNpx.Nov 18 2025, 12:15 PM

At the scale we are talking, they won't make any dent in the stats.

It's about 0.5% difference in count of 250, which isn't a vast amount, but it's not nothing. And the ranking of the top-30-by-hits changes (at least 200/600 swap places, there are other shifts too, albeit not in the top 10). So I think it was worth spending a little time working on improving the query.

MatthewVernon mentioned this in T408062: FY 25/26 WE 5.4.7 Standardize thumbnail sizes.Nov 18 2025, 11:31 PM

MatthewVernon mentioned this in T372165: Reduce number of bucketsizes for MediaViewer.Nov 18 2025, 11:36 PM

bvibber mentioned this in T410556: Spike: Media Viewer bucket sizes smallest bucket is not the expected size if not in wgThumbnailSteps.Nov 19 2025, 8:12 PM

MatthewVernon mentioned this in T408715: Compile a list of "canonical" thumbnail sizes.Nov 20 2025, 4:28 PM

So 480 is quite common, but hasn't showed up in our search. I thought it might be instructive to check referer:

select referer, count(*) as hits from wmf.webrequest where webrequest_source='upload' and year=2025 and month=10 and day=24 and hour=10 and http_status='200' and uri_path like '/wikipedia/%/thumb/%' and regexp_extract(uri_path, '([0-9]+)px[^/]+$')='480' group by referer order by hits desc LIMIT 10;

Gives us

referer	hits
https://en.wikipedia.org/	664267
https://de.wikipedia.org/	159180
https://ru.wikipedia.org/	143781
https://www.ecosia.org/	143604
https://ja.wikipedia.org/	90948
https://fr.wikipedia.org/	88355
https://it.wikipedia.org/	75985
https://es.wikipedia.org/	54362
https://pl.wikipedia.org/	44896
https://zh.wikipedia.org/	28721

(compared to a total of 1,698,043) Which makes me think that it's something on-wiki that is producing these 480px links... Any ideas? The top 10 paths by hit-count are all SVG files (which may be a red herring of course).

Very likely a popular gadget/css hardcoding the url. I investigate once I get my hands on a PC

Thanks! I've spent a fair chunk of time searching and have come up with nothing. My next stop is likely #no-stupid-questions...

I have solved the easy one, though: ecosia. If you image search on there (e.g. https://www.ecosia.org/images?q=cattle and find the wikipedia hit (about fourth row down), it's hard-coding the link to https://upload.wikimedia.org/wikipedia/commons/thumb/8/8c/Cow_(Fleckvieh_breed)_Oeschinensee_Slaunger_2009-07-07.jpg/480px-Cow_(Fleckvieh_breed)_Oeschinensee_Slaunger_2009-07-07.jpg (though that does bring about the question of where/how it's getting that from )

I haven't found anything in gadgets, etc. https://commons.wikimedia.org/w/index.php?title=Special:Search&limit=500&offset=0&ns8=1&search=%22480px-%22 (same in enwiki, etc.) I found a lot of weird stuff in meta but still all unrelated: https://meta.wikimedia.org/w/index.php?search=%22%2F480px-%22&title=Special%3ASearch&profile=advanced&fulltext=1&ns8=1

What I found extremely interesting though is that it's not one image. The top path gets only 2K hits in an hour:

spark-sql (default)> select uri_path, count(*) as hits from wmf.webrequest where webrequest_source='upload' and year=2025 and month=10 and day=24 and hour=10 and http_status='200' and uri_path like '/wikipedia/%/thumb/%' and regexp_extract(uri_path, '([0-9]+)px[^/]+$')='480' group by uri_path order by hits desc LIMIT 10;
uri_path	hits
/wikipedia/commons/thumb/8/82/Telegram_logo.svg/480px-Telegram_logo.svg.png	2430
/wikipedia/commons/thumb/2/2f/Google_2015_logo.svg/480px-Google_2015_logo.svg.png	2279
/wikipedia/commons/thumb/e/e4/Status_iucn3.1_LC_ru.svg/480px-Status_iucn3.1_LC_ru.svg.png	2139
/wikipedia/commons/thumb/9/96/Flag_of_the_United_States_%28DDD-F-416E_specifications%29.svg/480px-Flag_of_the_United_States_%28DDD-F-416E_specifications%29.svg.png	1946
/wikipedia/commons/thumb/a/a5/Flag_of_the_United_Kingdom_%281-2%29.svg/480px-Flag_of_the_United_Kingdom_%281-2%29.svg.png	1543
/wikipedia/commons/thumb/f/f3/Flag_of_Russia.svg/480px-Flag_of_Russia.svg.png	1493
/wikipedia/commons/thumb/3/32/Googleplex_HQ_%28cropped%29.jpg/480px-Googleplex_HQ_%28cropped%29.jpg	1485
/wikipedia/commons/thumb/c/c3/Flag_of_France.svg/480px-Flag_of_France.svg.png	1432
/wikipedia/commons/thumb/2/20/YouTube_2024.svg/480px-YouTube_2024.svg.png	1405

if it was a gadget or something, it would have been one image being hammered.

Samwalton9-WMF subscribed.Nov 24 2025, 1:50 PM

akosiaris subscribed.Nov 24 2025, 5:39 PM

Turnilo for the Telegram Logo (first hit in what @Ladsgroup ) says: Google Proxy as the ISP, in an staggering 85% of the cases. However, it sends those requests with no referrer.

@Ladsgroup, just to be consistent with what @MatthewVernon reported above, should your query have an AND for the referer to match %.wikipedia.org?

Novem_Linguae subscribed.Nov 24 2025, 8:12 PM

spark-sql (default)> select uri_path, count(*) as hits from wmf.webrequest where webrequest_source='upload' and year=2025 and month=10 and day=24 and hour=10 and http_status='200' and uri_path like '/wikipedia/%/thumb/%' and regexp_extract(uri_path, '([0-9]+)px[^/]+$')='480' AND referer like '%.wikipedia.org' group by uri_path order by hits desc LIMIT 10;
uri_path	hits
/wikipedia/commons/thumb/7/7e/Map_of_Fukuoka_Prefecture_Ja.svg/480px-Map_of_Fukuoka_Prefecture_Ja.svg.png	1
/wikipedia/commons/thumb/a/a5/Infobox_info_icon2.svg/480px-Infobox_info_icon2.svg.png	1
/wikipedia/commons/thumb/d/d5/Lynx-wikipedia.png/480px-Lynx-wikipedia.png	1
/wikipedia/commons/thumb/8/81/Haiku_R1_Beta_3_desktop_screenshot.png/480px-Haiku_R1_Beta_3_desktop_screenshot.png	1
/wikipedia/commons/thumb/7/75/Small_Pencil_Icon.svg/480px-Small_Pencil_Icon.svg.png	1
/wikipedia/commons/thumb/5/5d/Dooble_Showing_Dutch_Wikipedia.png/480px-Dooble_Showing_Dutch_Wikipedia.png	1
/wikipedia/commons/thumb/e/eb/Emoji_u1f33f.svg/480px-Emoji_u1f33f.svg.png	1
/wikipedia/commons/thumb/f/fc/MEPIS_logo.svg/480px-MEPIS_logo.svg.png	1
/wikipedia/commons/thumb/6/62/W3m-wikipedia.png/480px-W3m-wikipedia.png	1
/wikipedia/commons/thumb/1/10/TDA1%2C_ptc_catalyst.svg/480px-TDA1%2C_ptc_catalyst.svg.png	1
Time taken: 56.367 seconds, Fetched 10 row(s)
spark-sql (default)>

The query was wrong, the like should have an extra % at the end. Let me try again.

spark-sql (default)> select uri_path, count(*) as hits from wmf.webrequest where webrequest_source='upload' and year=2025 and month=10 and day=24 and hour=10 and http_status='200' and uri_path like '/wikipedia/%/thumb/%' and regexp_extract(uri_path, '([0-9]+)px[^/]+$')='480' AND referer like '%.wikipedia.org%' group by uri_path order by hits desc LIMIT 10;
uri_path	hits
/wikipedia/commons/thumb/e/e4/Status_iucn3.1_LC_ru.svg/480px-Status_iucn3.1_LC_ru.svg.png	2106
/wikipedia/commons/thumb/9/96/Flag_of_the_United_States_%28DDD-F-416E_specifications%29.svg/480px-Flag_of_the_United_States_%28DDD-F-416E_specifications%29.svg.png	1939
/wikipedia/commons/thumb/a/a5/Flag_of_the_United_Kingdom_%281-2%29.svg/480px-Flag_of_the_United_Kingdom_%281-2%29.svg.png	1532
/wikipedia/commons/thumb/f/f3/Flag_of_Russia.svg/480px-Flag_of_Russia.svg.png	1482
/wikipedia/commons/thumb/c/c3/Flag_of_France.svg/480px-Flag_of_France.svg.png	1400
/wikipedia/commons/thumb/e/e2/Flag_of_the_United_States_%28Pantone%29.svg/480px-Flag_of_the_United_States_%28Pantone%29.svg.png	1297
/wikipedia/commons/thumb/f/fa/Flag_of_the_People%27s_Republic_of_China.svg/480px-Flag_of_the_People%27s_Republic_of_China.svg.png	1285
/wikipedia/commons/thumb/b/ba/Flag_of_Germany.svg/480px-Flag_of_Germany.svg.png	1272
/wikipedia/commons/thumb/e/e2/White_House_ballroom_plan%2C_October_2025.svg/480px-White_House_ballroom_plan%2C_October_2025.svg.png	1254
/wikipedia/commons/thumb/a/a9/Flag_of_the_Soviet_Union.svg/480px-Flag_of_the_Soviet_Union.svg.png	1218
Time taken: 36.124 seconds, Fetched 10 row(s)

I picked a random path that was hit and looked the IP and basically looked at the previous and after requests at the same time but the same IP. I picked a IPv6 to reduce the chance of overlap.

The only request to the actual wikis at the same time was to one page, and then there were requests to rest api endpoints linked in that page, so for example: https://de.wikipedia.org/api/rest_v1/page/summary/Frankreich and then loading of the image with 480px px (https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/Flag_of_France.svg/480px-Flag_of_France.svg.png), the problem is that the endpoint is not providing the incorrect size (it provides https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/Flag_of_France.svg/langde-330px-Flag_of_France.svg.png). Maybe caches got expired? I checked it for this month and it still we get a lot of those (top hits are mostly flags now):

spark-sql (default)> select uri_path, count(*) as hits from wmf.webrequest where webrequest_source='upload' and year=2025 and month=11 and day=24 and hour=10 and http_status='200' and uri_path like '/wikipedia/%/thumb/%' and regexp_extract(uri_path, '([0-9]+)px[^/]+$')='480' AND referer like '%.wikipedia.org%' group by uri_path order by hits desc LIMIT 10;
uri_path	hits
/wikipedia/commons/thumb/9/96/Flag_of_the_United_States_%28DDD-F-416E_specifications%29.svg/480px-Flag_of_the_United_States_%28DDD-F-416E_specifications%29.svg.png	2276
/wikipedia/commons/thumb/e/e4/Status_iucn3.1_LC_ru.svg/480px-Status_iucn3.1_LC_ru.svg.png	1970
/wikipedia/commons/thumb/c/c3/Flag_of_France.svg/480px-Flag_of_France.svg.png	1710
/wikipedia/commons/thumb/b/ba/Flag_of_Germany.svg/480px-Flag_of_Germany.svg.png	1685
/wikipedia/commons/thumb/a/a5/Flag_of_the_United_Kingdom_%281-2%29.svg/480px-Flag_of_the_United_Kingdom_%281-2%29.svg.png	1666
/wikipedia/commons/thumb/f/f3/Flag_of_Russia.svg/480px-Flag_of_Russia.svg.png	1635
/wikipedia/commons/thumb/a/a9/Flag_of_the_Soviet_Union.svg/480px-Flag_of_the_Soviet_Union.svg.png	1466
/wikipedia/commons/thumb/f/fa/Flag_of_the_People%27s_Republic_of_China.svg/480px-Flag_of_the_People%27s_Republic_of_China.svg.png	1436
/wikipedia/commons/thumb/c/cf/GraphyArchy_-_Wikipedia_00185-cropped.jpg/480px-GraphyArchy_-_Wikipedia_00185-cropped.jpg	1369
/wikipedia/commons/thumb/e/e2/Flag_of_the_United_States_%28Pantone%29.svg/480px-Flag_of_the_United_States_%28Pantone%29.svg.png	1343