No thumbnail available at https://commons.wikimedia.org/wiki/File:Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg
Request served via cp3079 cp3079, Varnish XID 691707132 Upstream caches: cp3079 int Error: 429, Too Many Requests at Sun, 20 Apr 2025 16:16:03 GMT
Please add Thumbor to thumbnail related tasks, otherwise no developer will find such tasks.
This file likely exceeds the Debian-provided resource limits for ImageMagick. It seems like it's failing fast, so it's likely exceeding them on load rather than while attempting to render the thumbnail. T220171: Generate thumbnails of large JPGs with VIPS is the solution.
https://commons.wikimedia.org/wiki/File:Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg is 28,674 × 20,246 pixels (580.53 Megapixel), but we have other files with similar resolution where there is no issue with thumbnails: https://commons.wikimedia.org/wiki/File:Daedongyeojido-full.jpg 17,837 × 30,000 pixels, 535.11 Megapixel; File:Van Gogh - Starry Night - Google Art Project.jpg 30,000 × 23,756 pixels, 712.68 Megapixel; https://commons.wikimedia.org/wiki/File:Pieter_Bruegel_the_Elder_-_The_Tower_of_Babel_(Vienna)_-_Google_Art_Project_-_edited.jpg 30,000 × 21,952 pixels, 658.56 Megapixel, and more generally images in https://commons.wikimedia.org/wiki/Category:Gigapixel_images_from_the_Google_Art_Project
Hm.... Trying to load the original https://upload.wikimedia.org/wikipedia/commons/c/c5/Himalaya%2C_Indian_Atlas%2C_sheet_66_%2815219000%29.jpg in Firefox ends up with an Image corrupt or truncated. error in the console. Trying to load the URL with my test Thumbor HTTP loader gives errors about HTTP Content-Length. identify -verbose takes a fairly long time to complete but does so without errors.
$ identify -verbose Himalaya,_Indian_Atlas,_sheet_66_\(15219000\).jpg Image: Filename: Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg Permissions: rw-r--r-- Format: JPEG (Joint Photographic Experts Group JFIF format) Mime type: image/jpeg Class: DirectClass Geometry: 28674x20246+0+0 Resolution: 72x72 Print size: 398.25x281.194 Units: PixelsPerInch Colorspace: sRGB Type: TrueColor Base type: Undefined Endianness: Undefined Depth: 8-bit Channels: 3.0 Channel depth: Red: 8-bit Green: 8-bit Blue: 8-bit Channel statistics: Pixels: 580533804 Red: min: 0 (0) max: 255 (1) mean: 183.913 (0.721227) median: 205 (0.803922) standard deviation: 59.024 (0.231467) kurtosis: -1.02395 skewness: -0.58938 entropy: 0.883561 Green: min: 0 (0) max: 255 (1) mean: 179.205 (0.702765) median: 201 (0.788235) standard deviation: 57.341 (0.224867) kurtosis: -0.975416 skewness: -0.615138 entropy: 0.884332 Blue: min: 0 (0) max: 255 (1) mean: 165.105 (0.647472) median: 186 (0.729412) standard deviation: 52.5198 (0.20596) kurtosis: -0.795282 skewness: -0.686176 entropy: 0.879595 Image statistics: Overall: min: 0 (0) max: 255 (1) mean: 176.074 (0.690488) median: 197.333 (0.773856) standard deviation: 56.2949 (0.220764) kurtosis: -0.931548 skewness: -0.630231 entropy: 0.882496 Rendering intent: Perceptual Gamma: 0.454545 Chromaticity: red primary: (0.64,0.33,0.03) green primary: (0.3,0.6,0.1) blue primary: (0.15,0.06,0.79) white point: (0.3127,0.329,0.3583) Matte color: grey74 Background color: white Border color: srgb(223,223,223) Transparent color: black Interlace: JPEG Intensity: Undefined Compose: Over Page geometry: 28674x20246+0+0 Dispose: Undefined Iterations: 0 Compression: JPEG Quality: 98 Orientation: Undefined Profiles: Profile-icc: 672 bytes Properties: date:create: 2025-04-24T10:45:42+00:00 date:modify: 2023-08-07T21:28:26+00:00 date:timestamp: 2025-04-24T11:22:37+00:00 icc:copyright: Public Domain icc:description: GIMP built-in sRGB icc:manufacturer: GIMP icc:model: sRGB jpeg:colorspace: 2 jpeg:sampling-factor: 1x1,1x1,1x1 signature: a5b27e9b6f10edbd54b2a08e17e2f8a5dfc7c4f6998883fa723df749029f9781 Artifacts: verbose: true Tainted: False Filesize: 267.593MiB Number pixels: 580.534M Pixel cache type: Memory Pixels per second: 42.8261MP User time: 13.450u Elapsed time: 0:14.555 Version: ImageMagick 7.1.1-47 Q16-HDRI x86_64 22763 https://imagemagick.org
I was able to get my test Thumbor instance to thumbnail the file by downloading it with wget and using the file loader. I'm not sure if the original is corrupt in some way or if this is a SRE-swift-storage issue.
https://commons.wikimedia.org/wiki/File:Rembrandt_-_The_Abduction_of_Europa_-_Google_Art_Project.jpg thumbnails failed, but https://commons.wikimedia.org/wiki/File:Rembrandt_Harmensz._van_Rijn_-_The_Abduction_of_Europa_-_Google_Art_Project.jpg OK. Same resolution: 30,000 × 23,413 pixels, 702.39 Megapixels
https://upload.wikimedia.org/wikipedia/commons/thumb/archive/c/c1/20250428201709%21Rembrandt_-_The_Abduction_of_Europa_-_Google_Art_Project.jpg/250px-Rembrandt_-_The_Abduction_of_Europa_-_Google_Art_Project.jpg shows a different error:
Request served via cp3081 cp3081, Varnish XID 639274738 Upstream caches: cp3081 int Error: 500, Internal Server Error at Mon, 28 Apr 2025 20:27:25 GMT
I reduced the size to 20,000 × 15,609 pixels, 312.18 Megapixels, and now it works.
That last URL is an archive URL, which I wouldn't generally expect to work (they're for deleted-by-admin objects).
I downloaded Himalaya,_Indian_Atlas,_sheet_66_\(15219000\).jpg from both clusters and checked they were identical images, and they are (md5sum 5eef31842c22488b82f0cd19c7a2a21e), so I am confident this is a thumbor issue with the image rather than an inconsistency in swift. And it is a pretty huge image!
Same as my post above: https://commons.wikimedia.org/wiki/File:Vincent_van_Gogh_-_La_Chambre_%C3%A0_coucher_-_Google_Art_Project,_Amsterdam.jpg
Temporarily reducing size to 20,000 × 15,869 pixels, 317.38 Megapixels, makes it works.
Will need someone with logstash access to go fishing in that case, since I can't reproduce locally.
Same error from other big files: https://commons.wikimedia.org/wiki/File:India_and_adjacent_countries,_1917.jpg
https://commons.wikimedia.org/wiki/File:India_and_adjacent_countries,_1907_(13441003).jpg and https://commons.wikimedia.org/wiki/File:1915_India_and_Adjacent_Countries_by_Survey_of_India.jpg
Should I open another report? It looks like identical to me.
https://commons.wikimedia.org/wiki/File:1915_India_and_Adjacent_Countries_by_Survey_of_India.jpg looks to me to have a complete set of thumbnails.
https://commons.wikimedia.org/wiki/File:India_and_adjacent_countries,_1917.jpg is identical in both swift clusters, and a very large image (thumbnailing it on my laptop uses around 6G of RAM)
https://commons.wikimedia.org/wiki/File:India_and_adjacent_countries,_1907_(13441003).jpg is identical in both swift clusters, and a very large image (thumbnailing it on my laptop uses around 5G of RAM)
We may have an idea of what is happening. It seems to be an issue with the imagemagick memory limits in the thumbor backends, but we need to do a little more testing to confirm.
Change #1171982 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] thumbor: Lower thumbor_workers, more memory
Change #1171996 had a related patch set uploaded (by Hnowlan; author: Hnowlan):
[operations/deployment-charts@master] thumbor: change haproxy load balancing algorithm to leastconn
Change #1171982 merged by jenkins-bot:
[operations/deployment-charts@master] thumbor: Lower thumbor_workers, more memory
We have rolled out a first change that has fixed thumbnail gen for the India map images linked above, at least for the couple of thumbnail sizes I checked.
We lowered the number of thumbor workers per pod, but allocated them more memory, which in turn allows more processing memory to the imagemagick policy.
Thumbor monitoring graphs look ok, the processing times going up is expected since the workers are now processing bigger images they couldn't before.
We have another change we want to deploy a little bit later that changes how we distribute the work between the workers inside of a thumbor pod that should help further.
Change #1171996 merged by jenkins-bot:
[operations/deployment-charts@master] thumbor: change haproxy load balancing algorithm to leastconn
I've checked each of the images mentioned in this ticket and it appears that all now can be thumbnailled successfully at at least some sizes. Our second change appears to have improved overall stability somewhat also. This change will not completely resolve issues with very large images, as there are still upper bounds to what thumbor can handle. Feel free to try to upload larger versions of images in the mean time though.
I think I found a way to fix that. Saving in with baseline instead of progressive, and then there is no display issue even with 1423.52 Megapixels: https://commons.wikimedia.org/wiki/File:Rembrandt_-_The_Abduction_of_Europa_-_Google_Art_Project.jpg