Page MenuHomePhabricator
Create Task
Maniphest T219569

libvips thumbnail generation fails for TIFF files with invalid ICC profiles
Closed, ResolvedPublic
Actions

Description

vips is unable to output thumbnails if the input file has an invalid ICC profile.

Known affected files

https://upload.wikimedia.org/wikipedia/commons/thumb/2/28/Julia_Margaret_Cameron_-_Queen_of_the_May_-_1984.166_-_Cleveland_Museum_of_Art.tif/lossy-page1-364px-Julia_Margaret_Cameron_-_Queen_of_the_May_-_1984.166_-_Cleveland_Museum_of_Art.tif.jpg
Request from x.x.x.x via cp1082 cp1082, Varnish XID 389143731
Error: 429, Too Many Requests at Thu, 28 Mar 2019 22:04:48 GMT

`CommandError: (['/usr/bin/vips', 'shrink', '/srv/thumbor/tmp/thumbor@8818/tmpE_GnDj', '/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png', '20', '20'], '', 'vipspng: profile \'icc\': 0h: PCS illuminant is not D50\nvips2png: unable to write "/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png"\n', 1)`

Details

SubjectRepoBranchLines +/-
Upgrade to 2.7operations/debs/python-thumbor-wikimediamaster+111 -29
engine.vips: fall back to tinyrgb when libpng complainsoperations/software/thumbor-pluginsmaster+58 -14
Customize query in gerrit

Related Objects
Search...

Event Timeline

Yann created this task.Mar 28 2019, 10:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 28 2019, 10:06 PM
Aklapper renamed this task from Thumbnail error for TIFF file to HTTP 429 error for all thumbnail sizes for 65MB TIFF file on Commons.Mar 29 2019, 9:00 AM
Aklapper added a project: Thumbor.

Thumbnails 🡒 Thumbor

Thibaut120094 added a subscriber: Thibaut120094.Apr 2 2019, 6:58 PM

Sometimes it shows a 500 internal server error instead of 429.

Request from x.x.x.x via cp1076 cp1076, Varnish XID 37636667
Error: 500, Internal Server Error at Tue, 02 Apr 2019 18:56:11 GMT

• Gilles added a subscriber: • Gilles.Apr 2 2019, 8:13 PM

It's an error coming from vips, about an ICC profile this file contains:

CommandError: (['/usr/bin/vips', 'shrink', '/srv/thumbor/tmp/thumbor@8818/tmpE_GnDj', '/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png', '20', '20'], '', 'vipspng: profile \'icc\': 0h: PCS illuminant is not D50\nvips2png: unable to write "/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png"\n', 1)

This seems relevant: https://github.com/libvips/libvips/issues/1248

It does sound like the image could be tweaked (eg. convert to a different ICC profile) to avoid this, though.

Aklapper renamed this task from HTTP 429 error for all thumbnail sizes for 65MB TIFF file on Commons to HTTP 429 error for all thumbnail sizes for 65MB TIFF file on Commons, due to libvips.May 27 2019, 9:22 AM
Aklapper added a project: Upstream.
Steinsplitter added a subscriber: Steinsplitter.Dec 7 2019, 2:34 PM
In T219569#5079252, @Gilles wrote:

It's an error coming from vips, about an ICC profile this file contains:

CommandError: (['/usr/bin/vips', 'shrink', '/srv/thumbor/tmp/thumbor@8818/tmpE_GnDj', '/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png', '20', '20'], '', 'vipspng: profile \'icc\': 0h: PCS illuminant is not D50\nvips2png: unable to write "/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png"\n', 1)

This seems relevant: https://github.com/libvips/libvips/issues/1248

It does sound like the image could be tweaked (eg. convert to a different ICC profile) to avoid this, though.

https://commons.wikimedia.org/wiki/File:Special_Collections_University_of_Amsterdam_-_Iconographia_Zoologica_-_UBA01_IZ1207714_001_-_Gubernatrix_cristata_Vieillot,_1817_-_P._Oudart_-_G._Engelmann_-_Print_-_1825-1834.tif as well?

AntiCompositeNumber renamed this task from HTTP 429 error for all thumbnail sizes for 65MB TIFF file on Commons, due to libvips to libvips thumbnail generation fails for TIFF files with invalid ICC profiles.Apr 29 2020, 10:38 PM
AntiCompositeNumber updated the task description. (Show Details)
AntiCompositeNumber merged a task: T241692: 500 Internal error when accessing the thumbnail of a 50MB TIFF file on Commons.
AntiCompositeNumber added subscribers: Masumrezarock100, Fuzheado, Tgr and 3 others.
AntiCompositeNumber added a parent task: T64466: TIFF handling (tracking).Apr 29 2020, 10:41 PM
AntiCompositeNumber added a comment.Apr 29 2020, 11:11 PM

The recommended fix https://github.com/libvips/libvips/issues/311 is to shrink into a temporary vips file (.v), then use vips icc_transform tempfile.v outfile.png /path/to/profiles/sRGB.icm to output the PNG. I confirmed locally that this method produces thumbnails where vips shrink... alone did not.

The downside of this method is that it requires writing to another temporary file. This could be avoided by using the Python bindings for libvips or vipsthumbnail instead of vips, both of which likely have their own downsides. In any case, the color profile should only be changed when the embedded one is invalid.

AntiCompositeNumber added a comment.Apr 29 2020, 11:13 PM

It's not ideal to remove color profiles from the original TIFF file though. TIFF files are typically considered archival images, so it's important to keep them as original as possible.

AntiCompositeNumber removed a project: Upstream.Apr 29 2020, 11:17 PM
AntiCompositeNumber updated the task description. (Show Details)Apr 29 2020, 11:26 PM
AntiCompositeNumber merged a task: T231771: Thumbnail for 50MB TIFF file not shown: HTTP 429 error (due to profile info in TIFF file?).
AntiCompositeNumber added subscribers: Mbch331, Nemo_bis, TheDJ, Bdijkstra.
Aklapper added a comment.Apr 30 2020, 8:28 AM

Why was Upstream removed from this task?

AntiCompositeNumber added a project: Upstream.Apr 30 2020, 5:02 PM
AntiCompositeNumber moved this task from Backlog to Upstream on the Thumbor board.

It looked to me like vips was only designed to perform one operation at a time, and thus can not convert to a working ICC profile and shrink the image in one command. Looking over everything again, I'm now less sure of that position, so I've opened https://github.com/libvips/libvips/issues/1635 and re-added the tags.

AntiCompositeNumber added a comment.EditedApr 30 2020, 5:55 PM

Apparently the issue is the intended operation of the library libvips uses to write PNGs. The suggestion from libvips is indeed to convert to srgb before writing to PNG. They also suggest using vipsthumbnail instead of vips.

gerritbot added a comment.May 2 2020, 5:36 PM

Change 593920 had a related patch set uploaded (by AntiCompositeNumber; owner: AntiCompositeNumber):
[operations/software/thumbor-plugins@master] engine.vips: fall back to tinyrgb when libpng complains

https://gerrit.wikimedia.org/r/593920

gerritbot added a project: Patch-For-Review.May 2 2020, 5:36 PM
gerritbot added a comment.May 4 2020, 11:24 AM

Change 593920 merged by Gilles:
[operations/software/thumbor-plugins@master] engine.vips: fall back to tinyrgb when libpng complains

https://gerrit.wikimedia.org/r/593920

• Gilles claimed this task.May 4 2020, 11:25 AM
• Gilles triaged this task as Medium priority.
• Gilles removed a project: Patch-For-Review.May 6 2020, 3:22 PM
gerritbot added a comment.May 12 2020, 9:41 AM

Change 595891 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.7

https://gerrit.wikimedia.org/r/595891

gerritbot added a project: Patch-For-Review.May 12 2020, 9:41 AM
gerritbot added a comment.May 12 2020, 1:17 PM

Change 595891 merged by Ema:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.7

https://gerrit.wikimedia.org/r/595891

Stashbot mentioned this in T236240: Ghostscript outputs errors to stdout despite -q, preventing Thumbor from generating some thumbnails properly.May 12 2020, 1:50 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-12T13:50:55Z] <ema> thumbor2001: upgrade python-thumbor-wikimedia to 2.7-1+deb10u1 T252509 T219569 T236240

Stashbot mentioned this in T252509: thumbor: set Cache-Control ensuring cacheability on 404 responses.May 12 2020, 1:50 PM
Stashbot added a comment.May 12 2020, 1:54 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-12T13:54:21Z] <ema> thumbor2001: pool thumbor 2.7-1+deb10u1 for prod traffic T252509 T219569 T236240

Stashbot added a comment.May 12 2020, 2:00 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-12T14:00:48Z] <ema> thumbor2001: depool due to minor bug in 2.7-1+deb10u1 T252509 T219569 T236240

Stashbot added a comment.May 12 2020, 2:33 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-12T14:33:11Z] <ema> thumbor2001: upgrade python-thumbor-wikimedia to 2.8-1+deb10u1 T252509 T219569 T236240

Stashbot added a comment.May 12 2020, 2:39 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-12T14:39:57Z] <ema> rolling thumbor upgrade to 2.8-1+deb10u1 T252509 T219569 T236240

• Gilles closed this task as Resolved.May 12 2020, 2:55 PM

Fix confirmed on:

https://commons.wikimedia.org/wiki/File:Julia_Margaret_Cameron_-_Queen_of_the_May_-_1984.166_-_Cleveland_Museum_of_Art.tif
https://commons.wikimedia.org/wiki/File:Moivre_-_Doctrine_of_chances,_1761_-_722666.tif
https://commons.wikimedia.org/wiki/File:Special_Collections_University_of_Amsterdam_-_Iconographia_Zoologica_-_UBA01_IZ1207714_001_-_Gubernatrix_cristata_Vieillot,_1817_-_P._Oudart_-_G._Engelmann_-_Print_-_1825-1834.tif
https://commons.wikimedia.org/wiki/File:The_Rames%C3%AAum._The_remains_of_the_great_hypostyle_of_the_Temple_of_Rameses_II._The_figures_of_Christian_slaves_have_been_painted_on_some_of_the_columns_(NYPL_b10607452-80839).tiff

Just purge affected files and they should generate thumbnails fine.

Thanks again @AntiCompositeNumber for your contribution!

AntiCompositeNumber mentioned this in T133175: Specific Thumbnail generation for a broken invalid TIF file not working anymore.May 12 2020, 10:09 PM
AntiCompositeNumber mentioned this in T253368: Thumbnail generation fails (HTTP 503) for some sizes of File:Montreal_Skyline_from_Mont_Royal_raw.png.May 22 2020, 3:54 AM
AntiCompositeNumber mentioned this in T289759: PNG thumbnail has wrong colors for a huge (>2GB) TIFF image file (fixed in upstream vips 8.10).Aug 31 2021, 4:24 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 31 2021, 5:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL