Thumbor regularly errors on giant JPGs because ImageMagick runs out of memory attempting to generate thumbnails for them. We fixed the very same same problem for giant TIFFs and PNGs with VIPS, which is designed for low memory consumption and resizing of giant images.

Special care should be applied to reproduce the existing conditional sharpening applied with ImageMagick. If the same strategy as for TIFFs and PNGs is used, there will be an intermediary image roughly twice the size of the target. The conditional sharpening kicks in when the target size is at least 15% smaller than the original. With a VIPS intermediary step, it means it would only kick in when the image is at least 15% than the intermediary step. There might be edge cases where the sharpening won't kick in anymore as a result of the intermediary step.

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
thumbor: Add jpg to vips	Ladsgroup	operations/deployment-charts	master	+2 -2
thumbor: Use VIPS for large JPGs if the container starts with a	Ladsgroup	operations/deployment-charts	master	+3 -2
Set JPGs to be thumbnailed via VIPS if md5 starts with given config	Ladsgroup-claude	operations/software/thumbor-plugins	master	+125 -6
[DNM] Run VIPS on jpg unconditionally	Ladsgroup	operations/software/thumbor-plugins	master	+1 -88

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		JTweed-WMF	T431141 FY26-27 PP3.6: Image thumbnailing
Open		Cparle	T431718 PP3.6.1: Thumbor performance
Open		None	T218089 Thumbnail fails with 500 Internal Server Error for original image file with 241MB
Open		None	T200313 HTTP 500 error for some larger thumbnails of 14MB file File:Lençóis_Maranhenses_2018.jpg on Commons
In Progress		Ladsgroup	T220171 Generate thumbnails of large JPGs with VIPS
Stalled	Spike	None	T361888 [SPIKE] Determine best solution to solve out-of-memory errors on JPG thumbnail generation

Event Timeline

• Gilles created this task.Apr 5 2019, 6:10 AM

• Gilles renamed this task from Generating thumbnails of large JPGs with VIPS instead of ImageMagick to Generating thumbnails of large JPGs with VIPS.

• Gilles renamed this task from Generating thumbnails of large JPGs with VIPS to Generate thumbnails of large JPGs with VIPS.

AntiCompositeNumber mentioned this in T245904: HTTP 500 and 503 (Backend fetch failed) errors when viewing thumbnail with length or width > 8000 px.May 13 2020, 7:20 PM

AntiCompositeNumber added a parent task: T200313: HTTP 500 error for some larger thumbnails of 14MB file File:Lençóis_Maranhenses_2018.jpg on Commons.May 16 2020, 12:39 AM

AntiCompositeNumber moved this task from Backlog to Format support on the Thumbor board.Aug 13 2020, 4:37 AM

AntiCompositeNumber mentioned this in T300735: Specific file inaccessible at highest resolution.Feb 2 2022, 4:10 PM

Don-vip subscribed.Oct 9 2023, 8:13 PM

MarkTraceur added a project: Structured-Data-Backlog (Current Work).Mar 4 2024, 5:34 PM

MarkTraceur moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.

MarkTraceur mentioned this in T361888: [SPIKE] Determine best solution to solve out-of-memory errors on JPG thumbnail generation.Apr 4 2024, 9:02 PM

MarkTraceur moved this task from Ready for Estimation to Blocked on the Structured-Data-Backlog (Current Work) board.May 29 2024, 4:23 PM

• AUgolnikova-WMF edited projects, added: Structured-Data-Backlog; removed: Structured-Data-Backlog (Current Work).Jul 8 2024, 4:40 PM

• AUgolnikova-WMF moved this task from Triage to Multimedia on the Structured-Data-Backlog board.Jul 10 2024, 3:13 PM

Changing status and priority to reflect the 6 years without a single comment in this task.

This isn't really stalled, there's just no one working on feature development for Thumbor.[1] T290759 is unrelated to the use of Vips in Thumbor.

AntiCompositeNumber mentioned this in T392348: No thumbnail available for File:Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg - Error: 429, Too Many Requests.Apr 22 2025, 2:57 AM

AntiCompositeNumber added a parent task: T392348: No thumbnail available for File:Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg - Error: 429, Too Many Requests.

AntiCompositeNumber added a subtask: T361888: [SPIKE] Determine best solution to solve out-of-memory errors on JPG thumbnail generation.

AntiCompositeNumber changed the status of subtask T361888: [SPIKE] Determine best solution to solve out-of-memory errors on JPG thumbnail generation from Open to Stalled.Apr 22 2025, 3:00 AM

AntiCompositeNumber removed a parent task: T392348: No thumbnail available for File:Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg - Error: 429, Too Many Requests.Apr 24 2025, 11:29 AM

Ladsgroup subscribed.Apr 16 2026, 6:38 PM

Ladsgroup mentioned this in T431141: FY26-27 PP3.6: Image thumbnailing.Jul 3 2026, 7:10 PM

If i recall,

Image magick could handle huge baseline jpegs fine, but not progressive. When i tested vips, it was the same, which is why we only did vips for pngs.

Of course this was more than a decade ago, so who knows what might have changed.

@Bawolff Do you recall, before we switched to Thumbor, whether WMF used MediaWiki VipsScaler such that it applied so scaling jpegs? At least for the majority of jpegs, which are baseline and not progressive?

I always assumed yes, and that the decision to not use it at all for jpegs happened as part of the Thumbor rollout. Looking at old tasks such as T53275 and parent/sibling tasks, it seems we intentionally declined Vips for all JPEGs in 2013 due to not being memory constant for progressive jpegs. But.. perhaps that changed between then and the Thumbor rollout in 2017.

Why is it a reason against using Vips for jpeg, if it is better for the majority of jpegs, and (potentially) the same as ImageMagick for progressive jpegs?

JTweed-WMF added a parent task: T431718: PP3.6.1: Thumbor performance.Jul 9 2026, 2:28 PM

JTweed-WMF added a project: MediaWiki-Media-Platform-Team.Sep 2 2026, 9:42 AM

JTweed-WMF moved this task from Inbox (Untriaged) to FY2627 Q1 on the MediaWiki-Media-Platform-Team board.

JTweed-WMF raised the priority of this task from Low to High.Sep 2 2026, 2:50 PM

JTweed-WMF moved this task from FY2627 Q1 to To Verify on the MediaWiki-Media-Platform-Team board.

JTweed-WMF moved this task from To Verify to FY2627 Q1 on the MediaWiki-Media-Platform-Team board.

Change #1339064 had a related patch set uploaded (by Ladsgroup-claude; author: Ladsgroup-claude):

[operations/software/thumbor-plugins@master] [WIP] Set JPGs to be thumbnailed via VIPS if md5 starts with a

https://gerrit.wikimedia.org/r/1339064

gerritbot added a project: Patch-For-Review.Thu, Sep 10, 6:48 PM

Ladsgroup changed the task status from Stalled to In Progress.Thu, Sep 10, 6:58 PM

Ladsgroup claimed this task.

Ladsgroup moved this task from FY2627 Q1 to In Progress on the MediaWiki-Media-Platform-Team board.

Change #1341337 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/software/thumbor-plugins@master] [DNM] Run VIPS on jpg unconditionally

https://gerrit.wikimedia.org/r/1341337

Ladsgroup moved this task from In Progress to Code Review on the MediaWiki-Media-Platform-Team board.Mon, Sep 14, 10:54 PM

As you see, I put a DNM to run comparison tests between images we have in CI and the output. What is a bit worrying is that VIPS output in some cases is 6 times larger (in terms of bytes). In terms of similarity score (SSIM) so far, it's failing but not too bad (and under acceptable limits)

Change #1339064 merged by jenkins-bot:

[operations/software/thumbor-plugins@master] Set JPGs to be thumbnailed via VIPS if md5 starts with given config

https://gerrit.wikimedia.org/r/1339064

Change #1342312 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/deployment-charts@master] thumbor: Use VIPS for large JPGs if the container starts with a

https://gerrit.wikimedia.org/r/1342312

Change #1342312 merged by jenkins-bot:

[operations/deployment-charts@master] thumbor: Use VIPS for large JPGs if the container starts with a

https://gerrit.wikimedia.org/r/1342312

Change #1342316 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/deployment-charts@master] thumbor: Add jpg to vips

https://gerrit.wikimedia.org/r/1342316

Change #1342316 merged by jenkins-bot:

[operations/deployment-charts@master] thumbor: Add jpg to vips

https://gerrit.wikimedia.org/r/1342316

Now images that their hash start with a are being thumbnailed using vips and then IM. Example of image is https://commons.wikimedia.org/wiki/File:Kruger_National_Park_(ZA),_Giraffe_--_2024_--_0431.jpg

{"@timestamp":"2026-09-16T18:10:31.703Z","log.level":"debug","message":"[VIPS] Shrinking with command","ecs.version":"1.6.0","http":{"request":{"id":"0A401037:3F7F_0A404018:1010_6AAADB94_038A:0001"}},"labels":{"thumbor_version":"7.8.0"},"log":{"logger":"thumbor","origin":{"file":{"line":77,"name":"__init__.py"},"function":"debug"},"original":"[VIPS] Shrinking with command"},"process":{"name":"MainProcess","pid":9,"thread":{"id":139837363451776,"name":"MainThread"}},"service":{"node":{"name":"????"},"version":"0.1.0"},"url":{"original":"/wikipedia/commons/thumb/a/a8/Kruger_National_Park_%28ZA%29%2C_Giraffe_--_2024_--_0431.jpg/960px-Kruger_National_Park_%28ZA%29%2C_Giraffe_--_2024_--_0431.jpg"}}

In T220171#12101081, @Krinkle wrote:

@Bawolff Do you recall, before we switched to Thumbor, whether WMF used MediaWiki VipsScaler such that it applied so scaling jpegs? At least for the majority of jpegs, which are baseline and not progressive?

I always assumed yes, and that the decision to not use it at all for jpegs happened as part of the Thumbor rollout. Looking at old tasks such as T53275 and parent/sibling tasks, it seems we intentionally declined Vips for all JPEGs in 2013 due to not being memory constant for progressive jpegs. But.. perhaps that changed between then and the Thumbor rollout in 2017.

Why is it a reason against using Vips for jpeg, if it is better for the majority of jpegs, and (potentially) the same as ImageMagick for progressive jpegs?

At the time, it was the same for all types of jpegs. image magick (actually libjpeg) used low memory when converting baseline jpegs.

There didn't seem any benefit switching to libvips and there was concerns there might be mild quality degradation with vips so we just kept it with image magick.

Note: Back in the day for image magick to use low memory on jpegs, you had to pass a special argument -define jpeg:size=TARGET_SIZE_HERE . If thumbor isn't doing that, that might be why we had a regresison here.

Note: there is a complaint on discord about hatching on some thumbnails. The thumbnail in question had an md5 starting with a, so plausibly related to this.

Has there been a change in how Commons is processing images? I'm finding a sort of cross-hatching in thumbs recently.
[Image https://commons.wikimedia.org/wiki/File:Essex_Police_Car_54,_Canadian_Transportation_Museum,_Kingsville,_Ontario,_2026-10-04_03.jpg ]

It took me a bit but I do see that thumbnails look a bit sharper than the original:

Note the grass and the road.

I look into this a bit more.

Out of curiosity, did we measure what the improvement was in terms of memory and latency when using vips on jpegs?

I've been looking into p95 and p99 of thumbor in requests to containers starting with a vs b (literally a/b testing :P).

Here is the result:

(a)
year    month   day     hour    p95_time_firstbyte      n_requests
2026    10      7       0       4.392   36401
2026    10      7       1       4.273   36220
2026    10      7       2       3.9     35163
2026    10      7       3       4.093   36548
2026    10      7       4       4.284   37247
2026    10      7       5       4.303   38480
2026    10      7       6       4.286   38851
2026    10      7       7       4.281   44000
2026    10      7       8       3.9     46242
2026    10      7       9       3.7119999999999997      53268
2026    10      7       10      3.749   78098
2026    10      7       11      4.12    65948
2026    10      7       12      4.159   65646
2026    10      7       13      4.28    67695
2026    10      7       14      4.24    63359
2026    10      7       15      3.953   58738
2026    10      7       16      3.592   52118
2026    10      7       17      3.313   53709
2026    10      7       18      3.271   55780
2026    10      7       19      3.004   56197
2026    10      7       20      2.84    58060
2026    10      7       21      2.873   48210
2026    10      7       22      2.825   41091
2026    10      7       23      3.34    42142
Time taken: 161.365 seconds, Fetched 24 row(s)


(b)
year    month   day     hour    p95_time_firstbyte      n_requests
2026    10      7       0       4.293   50351
2026    10      7       1       4.006   51047
2026    10      7       2       3.377   48380
2026    10      7       3       3.711   47837
2026    10      7       4       4.272   50473
2026    10      7       5       4.276   54896
2026    10      7       6       3.949   59224
2026    10      7       7       4.265   59216
2026    10      7       8       3.9859999999999998      62594
2026    10      7       9       3.851   64146
2026    10      7       10      4.298   81124
2026    10      7       11      4.337   75794
2026    10      7       12      4.268   81206
2026    10      7       13      4.46    82028
2026    10      7       14      4.248   78987
2026    10      7       15      4.166   71886
2026    10      7       16      3.394   71028
2026    10      7       17      3.071   70516
2026    10      7       18      3.023   70246
2026    10      7       19      2.887   72799
2026    10      7       20      2.686   75036
2026    10      7       21      2.634   67111
2026    10      7       22      2.794   49236
2026    10      7       23      3.2720000000000002      47161
Time taken: 274.864 seconds, Fetched 24 row(s)

------------------
(a)
year    month   day     hour    p99_time_firstbyte      n_requests
2026    10      7       0       8.074   36401
2026    10      7       1       7.047   36220
2026    10      7       2       6.515   35163
2026    10      7       3       6.854   36548
2026    10      7       4       7.754   37247
2026    10      7       5       7.688   38480
2026    10      7       6       7.62    38851
2026    10      7       7       8.02    44000
2026    10      7       8       8.105   46242
2026    10      7       9       7.242   53268
2026    10      7       10      10.027  78098
2026    10      7       11      9.461   65948
2026    10      7       12      9.92    65646
2026    10      7       13      10.146  67695
2026    10      7       14      9.052   63359
2026    10      7       15      8.385   58738
2026    10      7       16      7.299   52118
2026    10      7       17      6.964   53709
2026    10      7       18      6.896   55780
2026    10      7       19      6.044   56197
2026    10      7       20      6.213   58060
2026    10      7       21      5.99    48210
2026    10      7       22      6.287   41091
2026    10      7       23      6.024   42142
Time taken: 280.261 seconds, Fetched 24 row(s)

(b)
year    month   day     hour    p99_time_firstbyte      n_requests
2026    10      7       0       7.914   50351
2026    10      7       1       6.907   51047
2026    10      7       2       6.313   48380
2026    10      7       3       6.898   47837
2026    10      7       4       7.327   50473
2026    10      7       5       7.352   54896
2026    10      7       6       7.075   59224
2026    10      7       7       8.158   59216
2026    10      7       8       8.454   62594
2026    10      7       9       7.469   64146
2026    10      7       10      11.357  81124
2026    10      7       11      10.63   75794
2026    10      7       12      9.711   81206
2026    10      7       13      10.92   82028
2026    10      7       14      9.228   78987
2026    10      7       15      9.345   71886
2026    10      7       16      7.108   71028
2026    10      7       17      6.768   70516
2026    10      7       18      6.4719999999999995      70246
2026    10      7       19      6.126   72799
2026    10      7       20      6.03    75036
2026    10      7       21      5.8469999999999995      67111
2026    10      7       22      6.327   49236
2026    10      7       23      6.346   47161
Time taken: 104.001 seconds, Fetched 24 row(s)

the average is slightly smaller but none are actually statistically significant (with both unpaired and paired t-tests)

Generate thumbnails of large JPGs with VIPSOpen, In Progress, HighPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Generate thumbnails of large JPGs with VIPS
Open, In Progress, HighPublic
Actions

Related Objects
Search...