Page MenuHomePhabricator

HTTP 500 and 503 (Backend fetch failed) errors when viewing thumbnail with length or width > 8000 px
Open, LowPublic

Description

At https://en.wikisource.org/wiki/Page:1582_Rhemes_New_Testament.pdf/13, click "Image" link to https://upload.wikimedia.org/wikipedia/commons/thumb/1/1b/1582_Rhemes_New_Testament.pdf/page13-5645px-1582_Rhemes_New_Testament.pdf.jpg

Request from [snip] via cp4025 frontend, Varnish XID 327105949
Upstream caches: cp4025 int
Error: 500, Internal Server Error at Sat, 22 Feb 2020 03:59:27 GMT

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Tried again, got 503:

Request from [snip] via cp4025 frontend, Varnish XID 325012919
Error: 503, Backend fetch failed at Sat, 22 Feb 2020 04:01:38 GMT
Aklapper renamed this task from 500 error when viewing a file to HTTP 503 Error (Backend catch failed) when viewing a specific PDF thumbnail on Wikisource.Feb 22 2020, 1:21 PM

Removing MediaWiki-File-management as I don't see a bug in MediaWiki core code; removing Traffic and Ops as I see no relation to traffic.
Adding SRE-swift-storage as this sounds like a backend storage issue. Adding Thumbor which creates thumbnails.

Testing locally, it appears that the issue has to do with ImageMagick size limits. MediaWiki reports that the PDF is 5,645 × 8,685 pixels, but page 13 is actually 6,108 × 8,918 pixels. To prevent rounding errors, we set the ImageMagick JPEG size hint to twice the target resolution. For this thumbnail, that's 11290x16486, so the width is above the Debian ImageMagick size limit of 16,000 px. That would explain the 500 error. On my machine with no ImageMagick policies configures, generation still fails, just with a SIGKILL instead of an error.

thumbor: ERROR: [ThreadPool] Failed to convert image convert: width or height exceeds limit `/tmp/tmppswAHz' @ error/cache.c/OpenPixelCache/3802.
convert: no images defined `jpg:-' @ error/convert.c/ConvertImageCommand/3258.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/thumbor/context.py", line 268, in _execute_in_foreground
    returned = operation()
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 334, in _load_results
    results = context.request.engine.read(image_extension, quality)
  File "/srv/thumbor-plugins/wikimedia_thumbor/engine/proxy/proxy.py", line 133, in read
    ret = self.__getattr__('read')(extension, quality)
  File "/srv/thumbor-plugins/wikimedia_thumbor/engine/__init__.py", line 40, in read
    return super(BaseWikimediaEngine, self).read(extension, quality)
  File "/srv/thumbor-plugins/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 332, in read
    raise ImageMagickException('Failed to convert image %s' % stderr)  # pragma: no cover
ImageMagickException: Failed to convert image convert: width or height exceeds limit `/tmp/tmppswAHz' @ error/cache.c/OpenPixelCache/3802.
convert: no images defined `jpg:-' @ error/convert.c/ConvertImageCommand/3258.

It would be helpful for someone with log access to check if this is actually the problem here, as well as to see what happened with the 503.

Generation of a 6,108 × 8,918 px thumbnail for this file suceeds because there is no resizing necessary. Unfortunately, ProofreadPage doesn't have that information, so it links to the smaller but broken image instead. I'm not sure if there's a good way to fix this on the Thumbor side other than T220171.

AntiCompositeNumber renamed this task from HTTP 503 Error (Backend catch failed) when viewing a specific PDF thumbnail on Wikisource to HTTP 500 and 503 (Backend fetch failed) errors when viewing thumbnail with length or width > 8000 px.May 13 2020, 7:22 PM