@Jdforrester-WMF @Catrope, as you both worked on this previously (for the Swedish Wikipedia at least), tagging you for your assistance as well. Thank you to all!

@taavi, noted! I'm new to Phabricator so didn't fully understand the implications of the assignment.

In T355914#9489729, @Ganesha811 wrote:

@Jdforrester-WMF @Catrope, as you both worked on this previously (for the Swedish Wikipedia at least), tagging you for your assistance as well. Thank you to all!

Hi there. This is a good thing to think about, but the last few times we've considered this it's been vetoed because of the risk of taking down the whole cluster due to the huge new spike of thumbnailing requests (because even though 250px is a 'standard', the vast majority of cached thumbnails are those from logged-out users' page requests, and so are only at the old standard size). Although small wikis can get away with this, the wikis with huge numbers of page views could overwhelm the cluster's ability to render images and take at least part of it down for all users.

Unfortunately I don't think anything has changed here, but perhaps SRE now think it's a burst we can cope with?

In T355914#9494311, @Superpes15 wrote:

I can process this in the next few hours (tomorrow at the latest)!

Please don't rush to try to implement this in config before SRE weigh in.

Uhm @Jdforrester-WMF I’m sorry, I didn’t find in the manuals that this required a SRE intervention! I obviously know that is a massive change, but perhaps I thought this was already performed in the past for other big projects (iirc - even if in this moment I don’t have the task numbers)! Then we’ll wait for SRE! Thanks :)

Given the chosen size is both non-standard (meaning it's not used on most large wikis) and not in the list of thumbnail sizes we pregenerate at upload time, I would imagine switching enwiki to this new thumbnail size would have a big impact on both the upload edge clusters and the backend object storage.

If I had to guess, we cannot do it without some work on pre-generating thumbnails; however I'll let the relevant teams comment on this.

I think the main issue is likely that we'll melt Thumbor if we just switch enwiki to 250, because 250 isn't a pregenerated size, and last time someone looked (T211661#8377883) only about 2% of requests were for that size. So I would assume that for the vast majority of images on enwiki we don't currently have a 250 thumb.

I wonder if we could arrange to gradually phase in the new size (maybe at the mw layer); that's rather outside my area of expertise, though.

We could have MW detect thumbnail transclusions with the default size, have a configuration setting for a 1/x ratio, then fire off a job that triggers a request to thumbor, and slowly increase the ratio over a couple of weeks ? Not sure how internal requests to the external thumbor url would work in terms of routing, but I'm sure someone else knows if that is possible.

And we have to remember it is more than just 250! Because of retina screens it is also 375 and 500. (replacing 220, 330, 440).

This is much messier and much hairier than it looks. And yes, without proper preparation this will bring down everything, not just thumbor. I put some random notes here:

• For the sake of reliability of the site, it's better to change the default everywhere not just English Wikipedia to avoid fragmenting thumbnails. It's been long-over due anyway.
• While we are here, there are two thumbnailing defaults, one for the articles that you want it changed but there is also thumbsize of categories and galleries which is currently 120px and I recommend bumping that to 250px too. That would eliminate thumb and cache fragmentation. There has been calls to increase that as well.
• To avoid melting thumbor, you can change the default in random subset of requests, let's say 1% and leave it be for a day and then bump it to 5%, leave it for a day, .... It should be down in span of a week at least. Not to mention the cache pollution problem that would be caused by it.
• That would save thumbor but swift, edge caches and network can also go down.
  • For swift, we really need to T211661: Automatically clean up unused thumbnails in Swift and this is a hard blocker (and important to do for other reasons as well). Currently we have 256TB of thumbnails, this will add a lot more. As I advocated before, we can just slowly delete all thumbnails and let the used ones regenerate to clean the Augean stables (It'll take months just to do this and that's the fastest and simplest solution)
  • Someone needs to look at the numbers and give an estimate to traffic team for the increased traffic to the upload cluster. That might also mean we need to increase the capacity of edge caches (we should start asking for that soon so it could be budgeted for the next FY, otherwise we would need to wait for a year). Also if the increase data flow too much, someone needs to look into upgrading the network. I/F have been upgrading codfw but maybe also should wait for the eqiad upgrade too? (I admit I haven't followed the maint work to know if it's increasing the capacity as well) Also making sure the links between edges and main dcs will have enough capacity specially considering the redundancy. The most likely failure scenario here is not that it will saturate the links the minute we deploy this but it's when a link goes down for the maintenance and then links get saturated.
• Obviously this needs to be added to the pregen thumb sizes first.
• Once that's done, I suggest getting rid of thumbsizes user preferences altogether and making the thumbsize wiki-wide instead.
• Random note: Why 250px? I suggest doing some research on user experience and pick one that
• (Optional): As I advocated before, there is a need to standardize thumbnails as well (even as a policy), for a really long, search in mobile and desktop showed a different sized thumbnails, search in https://wikipedia.org portal showed a completely different thumbnail too and CX a different one and related pages, as you guessed it, a different one.

One general note here is that last time it was asked (and it has been asked many times in many wikis). It was clear that we need to do a lot of work to make it deployed, but it doesn't get done but that's just kicking the can down the road since requests like this will come again.

Once that's done, I suggest getting rid of thumbsizes user preferences altogether and making the thumbsize wiki-wide instead.

I suggest we make the preference apply CSS sizing with browser side image scaling. With higher default sizes, retina screen variants and overal better image scaling in browsers these days, that might be acceptable, even for those who have this set differently now for accessibility reasons..

Obviously this needs to be added to the pregen thumb sizes first.

Specifically, we will require this for galleries, cause that is why those now often throw 429s.

we should start asking for that soon so it could be budgeted for the next FY

I think it is clear that there are quite a few related thumbnail (pre-)generation issues that could be solved with a focused effort.

Is there a pre-generated size around 250px that would be easier to switch to? We can attempt to develop a new consensus on English Wikipedia for a relatively close size, if that helps. Given the broad general support for this measure there, I don't think it would be a huge issue.

The pregen values are:

amir@amir-ThinkPad-P1-Gen-3:~/mediawiki-config/wmf-config$ grep -ri -A10 UploadThumbnailRenderMap
InitialiseSettings.php:'wgUploadThumbnailRenderMap' => [
InitialiseSettings.php-	'default' => [ 320, 640, 800, 1280 ],

Hmm, so the nearest is 320px? That's not particularly close, given the current is 220 and the consensus was for a change to 250. A proposal was made for a change to 300 px in the prior discussion and did not gain consensus. Let me know if the WMF would find it helpful for English Wikipedians to try and reach a consensus on 320 - I'm willing to start such a discussion. But if you are technically able to get us to 250, that would be preferable! Thank you all for your work on and attention to this.

In T355914#9501705, @Joe wrote:

Given the chosen size is both non-standard (meaning it's not used on most large wikis) and not in the list of thumbnail sizes we pregenerate at upload time

sv.wikipedia has 250px as the logged-out default thumb size. For logged-in users, 220px is not available to select as a user preference. So any Commons image that is used at svwiki must have a 250px thumb already generated.

In T355914#9510509, @Redrose64 wrote:

In T355914#9501705, @Joe wrote:

Given the chosen size is both non-standard (meaning it's not used on most large wikis) and not in the list of thumbnail sizes we pregenerate at upload time

sv.wikipedia has 250px as the logged-out default thumb size. For logged-in users, 220px is not available to select as a user preference. So any Commons image that is used at svwiki must have a 250px thumb already generated.

That is a drop in the bucket though.

At risk of scope creep... while we're here, it'd be nice to also re-examine gallery sizes. See this conversation on Wikitech-l :

• https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/R3ZN5DMGW55AHOQXXCZVQC5CULGQFG67/

Short version: 120px is outrageously small and not in keeping with how others do it. The default gallery size should be a lot larger - maybe something for a user study if we want details, but really any increase would be an improvement. If the same size as the "new" default thumbnail is picked, then there isn't even any need to do anything extra. Or alternatively, just match an existing size created by the thumbnailer.

I don't want to slow this task on upgrading the default thumbnail size down with More Stuff. On the other hand, if we're doing some fancy "regenerate all the images" for all Wikipedias anyway, this is probably the time to do it, so we don't have to do this twice? I'm happy to create a spinoff ticket if desired, but I suspect that these are deeply linked tasks, since they're gated by the same problem of regenerating the thumbnail images.

As a random rant, perhaps it is my myopia speaking, but I would really love to have bigger default sizes across Wikipedias for the images. In fact it is in my todo list to propose increase for default Infobox sizes on Ukrainian Wikipedia, and come to think of it making default thumbnail sizes bigger would be a boon too. That said I feel like desktop "improvements" are a thing that is making this a thing that would likely want people to stick with those inadequately smaller images as now you have to work with so much less width available.

As a side note let us also remember that those small ass images used to be physically bigger in the past. As in pixels were bigger, on an old monitor 20 years ago 250px is quite a big image. On a modern monitor it is a teeny-tiny colourful blob you have you squint to properly study.

In T355914#9527616, @SnowFire wrote:

At risk of scope creep... while we're here, it'd be nice to also re-examine gallery sizes. See this conversation on Wikitech-l :

• https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/R3ZN5DMGW55AHOQXXCZVQC5CULGQFG67/

Short version: 120px is outrageously small and not in keeping with how others do it. The default gallery size should be a lot larger - maybe something for a user study if we want details, but really any increase would be an improvement. If the same size as the "new" default thumbnail is picked, then there isn't even any need to do anything extra. Or alternatively, just match an existing size created by the thumbnailer.

I agree but that should be done after this to avoid melting thumbor :(

In T355914#9527640, @Base wrote:

As a random rant, perhaps it is my myopia speaking, but I would really love to have bigger default sizes across Wikipedias for the images. In fact it is in my todo list to propose increase for default Infobox sizes on Ukrainian Wikipedia, and come to think of it making default thumbnail sizes bigger would be a boon too.

Noting that we will change all wikis at once to avoid fragmenting thumb sizes. Not just enwiki.

In re https://phabricator.wikimedia.org/T355914#9505896:

@Ladsgroup, why don't the pre-generated thumbnail sizes match the pre-specified thumbnail sizes in https://en.wikipedia.org/wiki/Special:Preferences#mw-prefsection-rendering-files ? The pregen values you list above are 320, 640, 800, and 1280. The only options available to editors are 120, 150, 180, 200, 220, 250, 300, and 400. None of those match. Are we really never pregenerating the thumbnail sizes that get used in practice?

@WhatamIdoing Welcome to the mess of thumbnailing sizes in mediawiki. I wrote in more details in T211661#8377883. TLDR: That pregen sizes are the sizes you see under the image description page. e.g. (today's POTD in commons):
https://commons.wikimedia.org/wiki/File:Gilbert_Studios_photograph_of_Harriet_Jacobs.jpg

Those get pre-generated but the actual thumbnail sizes don't. OTOH, non-default thumbnail sizes are really rare and not really worth generating.

Edit: Except that it's not. It supposed to reflect that AFAIK. It doesn't even do that.

In T355914#9532098, @WhatamIdoing wrote:

Are we really never pregenerating the thumbnail sizes that get used in practice?

They almost always get created by human action (editing the wikitext) at a human pace. Generally those images have time enough to generate on the fly when the page is first viewed. Pre-gen is mostly useful for intermediate scaling (not used any longer since thumbor), galleries, MMV, and the MMV/file page download links. Usecases where fast loading is expected, or needed, but that aren't used oft enough that you can presume the image to have been generated previously. See also: T211661#9418466. Unfortunately even in those cases it's just a mess right now.

Right. It's a mess, the numbers need to be changed, and nobody will want to be responsible, e.g., for "taking away" the 320px and substituting the more immediately useful 300px, or for changing the prefs settings from 300px to 320px. That will require, at minimum, a series of meetings. A plausible temporary "fix" would be to run a one-time job that pregens a 250px (and probably 300px) copy of every image that's currently used, and then switch the wiki(s). After that, the images can all be processed at human scale, when they're added to the articles.

I'd guess that it's something on the order of 10 million images (half that, if we do enwiki only), which could take some time (2 months = ~5 million seconds), but we have worse alternatives.

To give a bit of historical perspective, using larger thumbnail was a frequent request a decade ago. From when I was in the team managing those topics (Platform Engineering) I went to write the Standardized thumbnails sizes request for comments with the intent of having it immediately declined and to be used as a reference to decline future requests (as I did in the name of the team on T43712#469014 ).

I remember hewiki wanted to get a larger default size and meanwhile relied on a template to fix the thumbsize to I think 250px (instead of whatever the default is). I am pretty sure Gilles did look at the thumbnails discrepancies between wikis and the issue it caused on the frontend cache and backend Swift storage.

This request is probably a dupe of T71973: Allow bigger thumbnail sizes in personal settings or of it is subtask T51357: Increase WMF cluster default image thumb size.

Another step in the process here would be to look at how the change would visually affect current layouts on enwiki and across wikis, as well as test and document any implications and concerns before proceeding with planning the change. Tagging with the Web team backlog to reflect the conversations happening around this.

@DTorsani-WMF and @JScherer-WMF - per @Ladsgroup suggestions, wondering if you both can offer any ux recommendations for recommended thumbnail sizing (250px, or higher). Seems like this would be beneficial to research if it may apply across all wikis, not only enwiki.

In T355914#9502268, @Ladsgroup wrote:

This is much messier and much hairier than it looks. And yes, without proper preparation this will bring down everything, not just thumbor. I put some random notes here:

• For the sake of reliability of the site, it's better to change the default everywhere not just English Wikipedia to avoid fragmenting thumbnails. It's been long-over due anyway.
• While we are here, there are two thumbnailing defaults, one for the articles that you want it changed but there is also thumbsize of categories and galleries which is currently 120px and I recommend bumping that to 250px too. That would eliminate thumb and cache fragmentation. There has been calls to increase that as well.
• To avoid melting thumbor, you can change the default in random subset of requests, let's say 1% and leave it be for a day and then bump it to 5%, leave it for a day, .... It should be down in span of a week at least. Not to mention the cache pollution problem that would be caused by it.
• That would save thumbor but swift, edge caches and network can also go down.
  • For swift, we really need to T211661: Automatically clean up unused thumbnails in Swift and this is a hard blocker (and important to do for other reasons as well). Currently we have 256TB of thumbnails, this will add a lot more. As I advocated before, we can just slowly delete all thumbnails and let the used ones regenerate to clean the Augean stables (It'll take months just to do this and that's the fastest and simplest solution)
  • Someone needs to look at the numbers and give an estimate to traffic team for the increased traffic to the upload cluster. That might also mean we need to increase the capacity of edge caches (we should start asking for that soon so it could be budgeted for the next FY, otherwise we would need to wait for a year). Also if the increase data flow too much, someone needs to look into upgrading the network. I/F have been upgrading codfw but maybe also should wait for the eqiad upgrade too? (I admit I haven't followed the maint work to know if it's increasing the capacity as well) Also making sure the links between edges and main dcs will have enough capacity specially considering the redundancy. The most likely failure scenario here is not that it will saturate the links the minute we deploy this but it's when a link goes down for the maintenance and then links get saturated.
• Obviously this needs to be added to the pregen thumb sizes first.
• Once that's done, I suggest getting rid of thumbsizes user preferences altogether and making the thumbsize wiki-wide instead.
• Random note: Why 250px? I suggest doing some research on user experience and pick one that
• (Optional): As I advocated before, there is a need to standardize thumbnails as well (even as a policy), for a really long, search in mobile and desktop showed a different sized thumbnails, search in https://wikipedia.org portal showed a completely different thumbnail too and CX a different one and related pages, as you guessed it, a different one.

One general note here is that last time it was asked (and it has been asked many times in many wikis). It was clear that we need to do a lot of work to make it deployed, but it doesn't get done but that's just kicking the can down the road since requests like this will come again.

@RHo Thank you! Something to consider is also looking at thumbnail sizes in other areas. Mostly notably:

• In categories and galleries where it's currently 120px (e.g. https://commons.wikimedia.org/wiki/Alexanderplatz)
• In search result: Special:Search, mobile fronted, new vector, portals.

Thank you!

In T355914#9537231, @ovasileva wrote:

Another step in the process here would be to look at how the change would visually affect current layouts on enwiki and across wikis...

I think that enough editors are using larger images that most pages with serious problems have already been seen and fixed. I have seen few such problems, and I've been running 300px at enwiki for months.

One way to encourage a more significant review would be to put up a CentralNotice banner encouraging logged-in editors to opt-in to the larger size.

Providing engineering perspective on behalf of the WMF web team, I agree that if we want to make this change in English we should do it for all wikis. For mobile we're less concerned given images tend to be lazy loaded there. If you let us know a rough timeline we can get back to you on when we can do this.

Recommendation:

• Web team runs some synthetic tests with help from @Peter to get a sense of impact (if any).
• Web team and @Peter set up daily tests in production for an image heavy article on Vietnamese Wikipedia
• Enable it on Vietnamese Wikipedia first
• Allow web team 2-4 weeks to monitor impact of change on metrics (for example NavigationTiming)
• Roll out change everywhere.

Additional notes:

• Before enabling this everywhere however, we would suggest enabling it on a single wiki first so that we can see the impact on performance on a image-heavy article.
• We have a dashboard currently pointing at the Barack Obama article on English Wikipedia https://grafana.wikimedia.org/d/000000205/reading-web-performance?orgId=1
• Ideally we'd set similar metrics up for a smaller wiki (perhaps Vietnamese Wikipedia given I am aware this wiki loads more site styles and scripts than other projects). I recall @Peter and I were setting this up before but I'm not sure where we got to with that.

@Jdlrobson thanks for laying out this plan, and thank you to all the WMF staff who are thinking about how best to do this! It looks like changes to thumbnail sizes have been talked about for years, and I hope this time we can get get it done! Please let me know if there's anything that would be helpful in terms of engaging the English Wikipedia community.

I spoke with @DTorsani-WMF, @RHo and others on the WMF design team on this. Assuming the change only affects thumbnails that are currently 220 and adding 30px to their width, we don't think there will be any adverse UX implications.

My biggest concern was for portrait-aspect ratio images on mobile. I was worried that wider thumbnails in info boxes, for example, would push content too far down the page. Increasing thumbnail size from 220px to 250px adds ~100px of vertical height to an image, which pushes 2 lines of content down "below the fold" of a page. This is a relatively minor impact, and so we are ok with the change.

Example with 220px

Example with 250px

@Jdlrobson, do you know if there has been any further progress on this? Has this change been added to the schedule or work plan of any teams or is it still being assessed? Thanks!

In T355914#9578211, @Jdlrobson wrote:

Providing engineering perspective on behalf of the WMF web team, I agree that if we want to make this change in English we should do it for all wikis. For mobile we're less concerned given images tend to be lazy loaded there. If you let us know a rough timeline we can get back to you on when we can do this.

Recommendation:

• Web team runs some synthetic tests with help from @Peter to get a sense of impact (if any).
• Web team and @Peter set up daily tests in production for an image heavy article on Vietnamese Wikipedia
• Enable it on Vietnamese Wikipedia first
• Allow web team 2-4 weeks to monitor impact of change on metrics (for example NavigationTiming)
• Roll out change everywhere.

Additional notes:

• Before enabling this everywhere however, we would suggest enabling it on a single wiki first so that we can see the impact on performance on a image-heavy article.
• We have a dashboard currently pointing at the Barack Obama article on English Wikipedia https://grafana.wikimedia.org/d/000000205/reading-web-performance?orgId=1
• Ideally we'd set similar metrics up for a smaller wiki (perhaps Vietnamese Wikipedia given I am aware this wiki loads more site styles and scripts than other projects). I recall @Peter and I were setting this up before but I'm not sure where we got to with that.

@SToyofuku-WMF to set up a spike around next steps on running the synthetic tests with @Peter

We are also considering implementing T360589: De-fragment thumbnail sizes in mediawiki to allow for improved storage and caching which would in turn enable SREs to change the default thumb size (or we can even bundle these two changes together). It'd be great if we can do synthetic tests these changes too.