Thumbnails -> Thumbor

Again here
Request from 88.182.x.x via cp1076 cp1076, Varnish XID 155781783
Error: 500, Internal Server Error at Thu, 04 Apr 2019 17:24:20 GMT

Original file ‎(9,102 × 12,994 pixels, file size: 88.01 MB, MIME type: image/jpeg); 118.27 Megapixels

2500px fails: https://upload.wikimedia.org/wikipedia/commons/thumb/2/22/A_Girl_Defending_Herself_against_Eros%2C_by_William-Adolphe_Bouguereau.jpg/2500px-A_Girl_Defending_Herself_against_Eros%2C_by_William-Adolphe_Bouguereau.jpg for https://commons.wikimedia.org/wiki/File:A_Girl_Defending_Herself_against_Eros,_by_William-Adolphe_Bouguereau.jpg

3000px fails: https://upload.wikimedia.org/wikipedia/commons/thumb/2/22/A_Girl_Defending_Herself_against_Eros%2C_by_William-Adolphe_Bouguereau.jpg/3000px-A_Girl_Defending_Herself_against_Eros%2C_by_William-Adolphe_Bouguereau.jpg

2000px works: https://upload.wikimedia.org/wikipedia/commons/thumb/2/22/A_Girl_Defending_Herself_against_Eros%2C_by_William-Adolphe_Bouguereau.jpg/2000px-A_Girl_Defending_Herself_against_Eros%2C_by_William-Adolphe_Bouguereau.jpg

ZoomViewer: flash/no flash fails: https://tools.wmflabs.org/zoomviewer/index.php?f=A%20Girl%20Defending%20Herself%20against%20Eros%2C%20by%20William-Adolphe%20Bouguereau.jpg&flash=no
No response from server iipsrv.fcgi

ZoomViewer is unrelated here

This is the error happening for the original report's image (Marie Antoinette): https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2019.04.05/logback?id=AWnsBY_9gtdVElVzN5xc&_g=h@44136fa

	Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/handler/images/images.py", line 571, in _load_results
    results, content_type = BaseHandler._load_results(self, context)
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 334, in _load_results
    results = context.request.engine.read(image_extension, quality)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 133, in read
    ret = self.__getattr__('read')(extension, quality)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 321, in read
    raise ImageMagickException('Failed to convert image %s' % stderr)  # pragma: no cover
ImageMagickException: Failed to convert image convert: DistributedPixelCache 'shared secret expected' @ error/distribute-cache.c/ConnectPixelCacheServer/210.
convert: cache resources exhausted `/srv/thumbor/tmp/thumbor@8822/tmpQr6cpJ' @ error/cache.c/OpenPixelCache/3945.
convert: no images defined `jpg:-' @ error/convert.c/ConvertImageCommand/3258.

The more recent file you've reported (A girl defending herself) seems to hit the same issue: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2019.04.05/logback?id=AWnsCOAUsODVk2E3RPgG&_g=h@44136fa

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/handler/images/images.py", line 571, in _load_results
    results, content_type = BaseHandler._load_results(self, context)
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 334, in _load_results
    results = context.request.engine.read(image_extension, quality)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 133, in read
    ret = self.__getattr__('read')(extension, quality)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 321, in read
    raise ImageMagickException('Failed to convert image %s' % stderr)  # pragma: no cover
ImageMagickException: Failed to convert image convert: DistributedPixelCache 'shared secret expected' @ error/distribute-cache.c/ConnectPixelCacheServer/210.
convert: cache resources exhausted `/srv/thumbor/tmp/thumbor@8806/tmp7JMb3l' @ error/cache.c/OpenPixelCache/3945.
convert: no images defined `jpg:-' @ error/convert.c/ConvertImageCommand/3258.

It's either an upstream issue with ImageMagick (https://github.com/ImageMagick/ImageMagick/issues/876 sure sounds like the same thing), and/or those images require too much memory to resize. 128 megapixels isn't a trivial size to process.

The solution would probably be to have giant JPGs processed by VIPS instead of ImageMagick, just like giant TIFFs and PNGs are. But that's non-trivial work, as the historical sharpening to JPGs via ImageMagick would have to be matched by the VIPS processing, with VIPS using a different sharpening model: https://jcupitt.github.io/libvips/API/current/libvips-convolution.html#vips-sharpen

I wonder if this is a new issue, or why it doesn't seem to have been reported earlier.
So I made more tests.

https://commons.wikimedia.org/wiki/File:Alfred_Sisley_-_The_Road_from_Versailles_to_Saint-Germain_-_Google_Art_Project.jpg
Original file : 8,552 × 6,672 pixels, file size: 25.9 MB, MIME type: image/jpeg; 57.06 Megapixel
Thumbnails are ok for 2500px, 3000px, 4000px, but fails for 5000px https://upload.wikimedia.org/wikipedia/commons/thumb/5/52/Alfred_Sisley_-_The_Road_from_Versailles_to_Saint-Germain_-_Google_Art_Project.jpg/5000px-Alfred_Sisley_-_The_Road_from_Versailles_to_Saint-Germain_-_Google_Art_Project.jpg

https://commons.wikimedia.org/wiki/File:%C3%89douard_Manet,_The_Rue_Mosnier_with_Flags,_1878.jpg
Original file : 8,516 × 6,883 pixels, file size: 81.58 MB, MIME type: image/jpeg; 58.62 Megapixel
Thumbnails are ok for 2000px, 3000px, 4000px, and 5000px.
Compared with the above, this is weird: almost identical size in pixels, but the smaller in MB fails where the bigger works.

https://commons.wikimedia.org/wiki/File:Jean-Antoine_Watteau,_The_Italian_Comedians_-_Getty_Museum.jpg
Original file: 11,103 × 15,245 pixels, file size: 97.46 MB, MIME type: image/jpeg; 169.27 Megapixel
Thumbnails are ok for 2000px, 3000px, but fails for 3500px and bigger: https://upload.wikimedia.org/wikipedia/commons/thumb/c/c7/Jean-Antoine_Watteau%2C_The_Italian_Comedians_-_Getty_Museum.jpg/3500px-Jean-Antoine_Watteau%2C_The_Italian_Comedians_-_Getty_Museum.jpg

https://commons.wikimedia.org/wiki/File:Jean-Fran%C3%A7ois_Millet_-_L%27Homme_%C3%A0_la_houe_(1860-62).jpg
Original file: 8,479 × 6,884 pixels, file size: 14.39 MB, MIME type: image/jpeg; 58.37 Megapixel
OK upto 4000px, but fails for 5000px: https://upload.wikimedia.org/wikipedia/commons/thumb/9/93/Jean-Fran%C3%A7ois_Millet_-_L%27Homme_%C3%A0_la_houe_%281860-62%29.jpg/5000px-Jean-Fran%C3%A7ois_Millet_-_L%27Homme_%C3%A0_la_houe_%281860-62%29.jpg

https://commons.wikimedia.org/wiki/File:Jean-Joseph-Xavier_Bidauld_-_Vue_du_pont_et_de_la_ville_de_Cava,_royaume_de_Naples.jpg
Original file: ‎8,530 × 6,455 pixels, file size: 17.41 MB, MIME type: image/jpeg; 55.06 Megapixel
OK upto 4000px, but fails for 5000px: https://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Jean-Joseph-Xavier_Bidauld_-_Vue_du_pont_et_de_la_ville_de_Cava%2C_royaume_de_Naples.jpg/5000px-Jean-Joseph-Xavier_Bidauld_-_Vue_du_pont_et_de_la_ville_de_Cava%2C_royaume_de_Naples.jpg

https://commons.wikimedia.org/wiki/File:Renoir_-_Landscape_at_Pont-Aven,_1892.jpg
Original file: 6,780 × 4,908 pixels, file size: 32.27 MB, MIME type: image/jpeg; 33.28 Megapixel
OK upto 3000px, but fails for 3500px: https://upload.wikimedia.org/wikipedia/commons/thumb/c/c6/Renoir_-_Landscape_at_Pont-Aven%2C_1892.jpg/3500px-Renoir_-_Landscape_at_Pont-Aven%2C_1892.jpg

https://commons.wikimedia.org/wiki/File:Dreux_Bud%C3%A9_Master_-_The_Crucifixion_-_Google_Art_Project.jpg
Original file: 9,187 × 7,103 pixels, file size: 14.38 MB, MIME type: image/jpeg); 65.26 Megapixel
OK upto 4000px, but fails for 5000px: https://upload.wikimedia.org/wikipedia/commons/thumb/e/ee/Dreux_Bud%C3%A9_Master_-_The_Crucifixion_-_Google_Art_Project.jpg/5000px-Dreux_Bud%C3%A9_Master_-_The_Crucifixion_-_Google_Art_Project.jpg

So all these do not seem to be consistent. What did I miss?

If you want me to do more tests, please let me know.

It could be that the recent software update to Debian Stretch brought us to an ImageMagick version that's more sensitive to this problem, using more memory for these particular files that the version of IM we previously ran on Debian Jessie did. I believe that the upgrade to Debian Buster is due this year, which will also bring a newer version of IM. Hopefully it might improve this situation.

In the meantime, I would advise where possible to upload TIFFs instead of JPGs when dealing with high megapixel counts. I know it means that the thumbnails won't be sharpened, but at least you should get working thumbnails for that content...

OK, good to know. But these JPEG files are provided as it is by GLAMs, so converting them to TIFF would create bloated files.

This appears to be quite an old bug that has been reported before. It isn't so much that the source is large but that the requested thumbnail is large. The thumbnailer had limits on size/memory for generated thumbnails. All these files generate thumbnails no problem, as long as the thumbnails aren't too large.
A solution I have used in the past to to upload a 50% size version of an image for those who can't view the full-size one and when the zoom browser isn't working. It would be nice if the thumbnailer could generate smaller versions of any size, though probably not a pressing issue for Wikipedia.