Page MenuHomePhabricator

libvips thumbnail generation fails for TIFF files with invalid ICC profiles
Closed, ResolvedPublic

Description

vips is unable to output thumbnails if the input file has an invalid ICC profile.

Known affected files


https://upload.wikimedia.org/wikipedia/commons/thumb/2/28/Julia_Margaret_Cameron_-_Queen_of_the_May_-_1984.166_-_Cleveland_Museum_of_Art.tif/lossy-page1-364px-Julia_Margaret_Cameron_-_Queen_of_the_May_-_1984.166_-_Cleveland_Museum_of_Art.tif.jpg
Request from x.x.x.x via cp1082 cp1082, Varnish XID 389143731
Error: 429, Too Many Requests at Thu, 28 Mar 2019 22:04:48 GMT

`CommandError: (['/usr/bin/vips', 'shrink', '/srv/thumbor/tmp/thumbor@8818/tmpE_GnDj', '/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png', '20', '20'], '', 'vipspng: profile \'icc\': 0h: PCS illuminant is not D50\nvips2png: unable to write "/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png"\n', 1)`

Event Timeline

Yann created this task.Mar 28 2019, 10:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 28 2019, 10:06 PM
Aklapper renamed this task from Thumbnail error for TIFF file to HTTP 429 error for all thumbnail sizes for 65MB TIFF file on Commons.Mar 29 2019, 9:00 AM
Aklapper added a project: Thumbor.

Thumbnails 🡒 Thumbor

Sometimes it shows a 500 internal server error instead of 429.

Request from x.x.x.x via cp1076 cp1076, Varnish XID 37636667
Error: 500, Internal Server Error at Tue, 02 Apr 2019 18:56:11 GMT

Gilles added a subscriber: Gilles.Apr 2 2019, 8:13 PM

It's an error coming from vips, about an ICC profile this file contains:

CommandError: (['/usr/bin/vips', 'shrink', '/srv/thumbor/tmp/thumbor@8818/tmpE_GnDj', '/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png', '20', '20'], '', 'vipspng: profile \'icc\': 0h: PCS illuminant is not D50\nvips2png: unable to write "/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png"\n', 1)

This seems relevant: https://github.com/libvips/libvips/issues/1248

It does sound like the image could be tweaked (eg. convert to a different ICC profile) to avoid this, though.

Aklapper renamed this task from HTTP 429 error for all thumbnail sizes for 65MB TIFF file on Commons to HTTP 429 error for all thumbnail sizes for 65MB TIFF file on Commons, due to libvips.May 27 2019, 9:22 AM
Aklapper added a project: Upstream.

It's an error coming from vips, about an ICC profile this file contains:

CommandError: (['/usr/bin/vips', 'shrink', '/srv/thumbor/tmp/thumbor@8818/tmpE_GnDj', '/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png', '20', '20'], '', 'vipspng: profile \'icc\': 0h: PCS illuminant is not D50\nvips2png: unable to write "/srv/thumbor/tmp/thumbor@8818/tmpQUoG4w/vips_result.png"\n', 1)

This seems relevant: https://github.com/libvips/libvips/issues/1248

It does sound like the image could be tweaked (eg. convert to a different ICC profile) to avoid this, though.

https://commons.wikimedia.org/wiki/File:Special_Collections_University_of_Amsterdam_-_Iconographia_Zoologica_-_UBA01_IZ1207714_001_-_Gubernatrix_cristata_Vieillot,_1817_-_P._Oudart_-_G._Engelmann_-_Print_-_1825-1834.tif as well?

AntiCompositeNumber renamed this task from HTTP 429 error for all thumbnail sizes for 65MB TIFF file on Commons, due to libvips to libvips thumbnail generation fails for TIFF files with invalid ICC profiles.Apr 29 2020, 10:38 PM
AntiCompositeNumber updated the task description. (Show Details)

The recommended fix https://github.com/libvips/libvips/issues/311 is to shrink into a temporary vips file (.v), then use vips icc_transform tempfile.v outfile.png /path/to/profiles/sRGB.icm to output the PNG. I confirmed locally that this method produces thumbnails where vips shrink... alone did not.

The downside of this method is that it requires writing to another temporary file. This could be avoided by using the Python bindings for libvips or vipsthumbnail instead of vips, both of which likely have their own downsides. In any case, the color profile should only be changed when the embedded one is invalid.

It's not ideal to remove color profiles from the original TIFF file though. TIFF files are typically considered archival images, so it's important to keep them as original as possible.

Why was Upstream removed from this task?

AntiCompositeNumber moved this task from Backlog to Upstream on the Thumbor board.

It looked to me like vips was only designed to perform one operation at a time, and thus can not convert to a working ICC profile and shrink the image in one command. Looking over everything again, I'm now less sure of that position, so I've opened https://github.com/libvips/libvips/issues/1635 and re-added the tags.

AntiCompositeNumber added a comment.EditedApr 30 2020, 5:55 PM

Apparently the issue is the intended operation of the library libvips uses to write PNGs. The suggestion from libvips is indeed to convert to srgb before writing to PNG. They also suggest using vipsthumbnail instead of vips.

Change 593920 had a related patch set uploaded (by AntiCompositeNumber; owner: AntiCompositeNumber):
[operations/software/thumbor-plugins@master] engine.vips: fall back to tinyrgb when libpng complains

https://gerrit.wikimedia.org/r/593920

Change 593920 merged by Gilles:
[operations/software/thumbor-plugins@master] engine.vips: fall back to tinyrgb when libpng complains

https://gerrit.wikimedia.org/r/593920

Gilles claimed this task.May 4 2020, 11:25 AM
Gilles triaged this task as Medium priority.

Change 595891 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.7

https://gerrit.wikimedia.org/r/595891

Change 595891 merged by Ema:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.7

https://gerrit.wikimedia.org/r/595891

Mentioned in SAL (#wikimedia-operations) [2020-05-12T13:54:21Z] <ema> thumbor2001: pool thumbor 2.7-1+deb10u1 for prod traffic T252509 T219569 T236240

Mentioned in SAL (#wikimedia-operations) [2020-05-12T14:00:48Z] <ema> thumbor2001: depool due to minor bug in 2.7-1+deb10u1 T252509 T219569 T236240

Mentioned in SAL (#wikimedia-operations) [2020-05-12T14:33:11Z] <ema> thumbor2001: upgrade python-thumbor-wikimedia to 2.8-1+deb10u1 T252509 T219569 T236240

Mentioned in SAL (#wikimedia-operations) [2020-05-12T14:39:57Z] <ema> rolling thumbor upgrade to 2.8-1+deb10u1 T252509 T219569 T236240