Page MenuHomePhabricator

Thumbnail fails with 500 Internal Server Error for original JPG file with 241MB
Open, NormalPublic

Event Timeline

Yann created this task.Mar 12 2019, 12:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 12 2019, 12:08 PM
Yann updated the task description. (Show Details)Mar 12 2019, 12:08 PM
Aklapper renamed this task from Thumbnail fails with 500 Internal Server Error to Thumbnail fails with 500 Internal Server Error for original file with 241MB.Mar 12 2019, 12:27 PM

ZoomViewer is unrelated here

jijiki added a subscriber: jijiki.Apr 4 2019, 9:55 PM
Yann triaged this task as Normal priority.Apr 5 2019, 5:21 AM

If it is happening once, it is minor. But now it is happening repeatedly to several files, it is a problem.

Gilles added a subscriber: Gilles.Apr 5 2019, 5:58 AM

This is the error happening for the original report's image (Marie Antoinette): https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2019.04.05/logback?id=AWnsBY_9gtdVElVzN5xc&_g=h@44136fa

	Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/handler/images/images.py", line 571, in _load_results
    results, content_type = BaseHandler._load_results(self, context)
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 334, in _load_results
    results = context.request.engine.read(image_extension, quality)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 133, in read
    ret = self.__getattr__('read')(extension, quality)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 321, in read
    raise ImageMagickException('Failed to convert image %s' % stderr)  # pragma: no cover
ImageMagickException: Failed to convert image convert: DistributedPixelCache 'shared secret expected' @ error/distribute-cache.c/ConnectPixelCacheServer/210.
convert: cache resources exhausted `/srv/thumbor/tmp/thumbor@8822/tmpQr6cpJ' @ error/cache.c/OpenPixelCache/3945.
convert: no images defined `jpg:-' @ error/convert.c/ConvertImageCommand/3258.

The more recent file you've reported (A girl defending herself) seems to hit the same issue: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2019.04.05/logback?id=AWnsCOAUsODVk2E3RPgG&_g=h@44136fa

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/handler/images/images.py", line 571, in _load_results
    results, content_type = BaseHandler._load_results(self, context)
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 334, in _load_results
    results = context.request.engine.read(image_extension, quality)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 133, in read
    ret = self.__getattr__('read')(extension, quality)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 321, in read
    raise ImageMagickException('Failed to convert image %s' % stderr)  # pragma: no cover
ImageMagickException: Failed to convert image convert: DistributedPixelCache 'shared secret expected' @ error/distribute-cache.c/ConnectPixelCacheServer/210.
convert: cache resources exhausted `/srv/thumbor/tmp/thumbor@8806/tmp7JMb3l' @ error/cache.c/OpenPixelCache/3945.
convert: no images defined `jpg:-' @ error/convert.c/ConvertImageCommand/3258.

It's either an upstream issue with ImageMagick (https://github.com/ImageMagick/ImageMagick/issues/876 sure sounds like the same thing), and/or those images require too much memory to resize. 128 megapixels isn't a trivial size to process.

The solution would probably be to have giant JPGs processed by VIPS instead of ImageMagick, just like giant TIFFs and PNGs are. But that's non-trivial work, as the historical sharpening to JPGs via ImageMagick would have to be matched by the VIPS processing, with VIPS using a different sharpening model: https://jcupitt.github.io/libvips/API/current/libvips-convolution.html#vips-sharpen

Aklapper renamed this task from Thumbnail fails with 500 Internal Server Error for original file with 241MB to Thumbnail fails with 500 Internal Server Error for original JPG file with 241MB.Apr 5 2019, 11:39 AM
Yann added a comment.Apr 5 2019, 8:34 PM

I wonder if this is a new issue, or why it doesn't seem to have been reported earlier.
So I made more tests.

https://commons.wikimedia.org/wiki/File:Alfred_Sisley_-_The_Road_from_Versailles_to_Saint-Germain_-_Google_Art_Project.jpg
Original file : 8,552 × 6,672 pixels, file size: 25.9 MB, MIME type: image/jpeg; 57.06 Megapixel
Thumbnails are ok for 2500px, 3000px, 4000px, but fails for 5000px https://upload.wikimedia.org/wikipedia/commons/thumb/5/52/Alfred_Sisley_-_The_Road_from_Versailles_to_Saint-Germain_-_Google_Art_Project.jpg/5000px-Alfred_Sisley_-_The_Road_from_Versailles_to_Saint-Germain_-_Google_Art_Project.jpg

https://commons.wikimedia.org/wiki/File:%C3%89douard_Manet,_The_Rue_Mosnier_with_Flags,_1878.jpg
Original file : 8,516 × 6,883 pixels, file size: 81.58 MB, MIME type: image/jpeg; 58.62 Megapixel
Thumbnails are ok for 2000px, 3000px, 4000px, and 5000px.
Compared with the above, this is weird: almost identical size in pixels, but the smaller in MB fails where the bigger works.

https://commons.wikimedia.org/wiki/File:Jean-Antoine_Watteau,_The_Italian_Comedians_-_Getty_Museum.jpg
Original file: 11,103 × 15,245 pixels, file size: 97.46 MB, MIME type: image/jpeg; 169.27 Megapixel
Thumbnails are ok for 2000px, 3000px, but fails for 3500px and bigger: https://upload.wikimedia.org/wikipedia/commons/thumb/c/c7/Jean-Antoine_Watteau%2C_The_Italian_Comedians_-_Getty_Museum.jpg/3500px-Jean-Antoine_Watteau%2C_The_Italian_Comedians_-_Getty_Museum.jpg

https://commons.wikimedia.org/wiki/File:Jean-Fran%C3%A7ois_Millet_-_L%27Homme_%C3%A0_la_houe_(1860-62).jpg
Original file: 8,479 × 6,884 pixels, file size: 14.39 MB, MIME type: image/jpeg; 58.37 Megapixel
OK upto 4000px, but fails for 5000px: https://upload.wikimedia.org/wikipedia/commons/thumb/9/93/Jean-Fran%C3%A7ois_Millet_-_L%27Homme_%C3%A0_la_houe_%281860-62%29.jpg/5000px-Jean-Fran%C3%A7ois_Millet_-_L%27Homme_%C3%A0_la_houe_%281860-62%29.jpg

https://commons.wikimedia.org/wiki/File:Jean-Joseph-Xavier_Bidauld_-_Vue_du_pont_et_de_la_ville_de_Cava,_royaume_de_Naples.jpg
Original file: ‎8,530 × 6,455 pixels, file size: 17.41 MB, MIME type: image/jpeg; 55.06 Megapixel
OK upto 4000px, but fails for 5000px: https://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Jean-Joseph-Xavier_Bidauld_-_Vue_du_pont_et_de_la_ville_de_Cava%2C_royaume_de_Naples.jpg/5000px-Jean-Joseph-Xavier_Bidauld_-_Vue_du_pont_et_de_la_ville_de_Cava%2C_royaume_de_Naples.jpg

https://commons.wikimedia.org/wiki/File:Renoir_-_Landscape_at_Pont-Aven,_1892.jpg
Original file: 6,780 × 4,908 pixels, file size: 32.27 MB, MIME type: image/jpeg; 33.28 Megapixel
OK upto 3000px, but fails for 3500px: https://upload.wikimedia.org/wikipedia/commons/thumb/c/c6/Renoir_-_Landscape_at_Pont-Aven%2C_1892.jpg/3500px-Renoir_-_Landscape_at_Pont-Aven%2C_1892.jpg

https://commons.wikimedia.org/wiki/File:Dreux_Bud%C3%A9_Master_-_The_Crucifixion_-_Google_Art_Project.jpg
Original file: 9,187 × 7,103 pixels, file size: 14.38 MB, MIME type: image/jpeg); 65.26 Megapixel
OK upto 4000px, but fails for 5000px: https://upload.wikimedia.org/wikipedia/commons/thumb/e/ee/Dreux_Bud%C3%A9_Master_-_The_Crucifixion_-_Google_Art_Project.jpg/5000px-Dreux_Bud%C3%A9_Master_-_The_Crucifixion_-_Google_Art_Project.jpg

So all these do not seem to be consistent. What did I miss?

Yann added a comment.Apr 5 2019, 8:35 PM

If you want me to do more tests, please let me know.

Gilles added a comment.EditedApr 6 2019, 8:37 AM

It could be that the recent software update to Debian Stretch brought us to an ImageMagick version that's more sensitive to this problem, using more memory for these particular files that the version of IM we previously ran on Debian Jessie did. I believe that the upgrade to Debian Buster is due this year, which will also bring a newer version of IM. Hopefully it might improve this situation.

Gilles added a comment.Apr 6 2019, 8:39 AM

In the meantime, I would advise where possible to upload TIFFs instead of JPGs when dealing with high megapixel counts. I know it means that the thumbnails won't be sharpened, but at least you should get working thumbnails for that content...

Yann added a comment.Apr 6 2019, 12:13 PM

OK, good to know. But these JPEG files are provided as it is by GLAMs, so converting them to TIFF would create bloated files.

Colin added a subscriber: Colin.Apr 22 2019, 5:49 PM

This appears to be quite an old bug that has been reported before. It isn't so much that the source is large but that the requested thumbnail is large. The thumbnailer had limits on size/memory for generated thumbnails. All these files generate thumbnails no problem, as long as the thumbnails aren't too large.
A solution I have used in the past to to upload a 50% size version of an image for those who can't view the full-size one and when the zoom browser isn't working. It would be nice if the thumbnailer could generate smaller versions of any size, though probably not a pressing issue for Wikipedia.