Page MenuHomePhabricator
Create Task
Maniphest T193326

HTTP 429 for File:Smědavská hornatina, Sloupský potok, vodopád 01.jpg at all thumbnail sizes
Closed, ResolvedPublic
Actions

Description

obrazek.png (444×1 px, 17 KB)

Error details:

error: could not load image from https://upload.wikimedia.org/wikipedia/commons/thumb/2/20/Sm%C4%9Bdavsk%C3%A1_hornatina%2C_Sloupsk%C3%BD_potok%2C_vodop%C3%A1d_01.jpg/800px-Sm%C4%9Bdavsk%C3%A1_hornatina%2C_Sloupsk%C3%BD_potok%2C_vodop%C3%A1d_01.jpg
URL: https://cs.wikipedia.org/wiki/Sloupsk%C3%BD_potok_(p%C5%99%C3%ADtok_Sm%C4%9Bd%C3%A9)#/media/File:Sm%C4%9Bdavsk%C3%A1_hornatina,_Sloupsk%C3%BD_potok,_vodop%C3%A1d_01.jpg
user agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
screen size: 1280x720
canvas size: 1263x522
image size: 2304x3072
thumbnail size: CSS: 392x522, screen width: 490, real width: 800

Details

SubjectRepoBranchLines +/-
Upgrade to 2.9operations/debs/python-thumbor-wikimediamaster+197 -53
engine.imagemagick: Catch pyexiv2.ExifValueError alsooperations/software/thumbor-pluginsmaster+21 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

matej_suchanek created this task.Apr 28 2018, 3:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 28 2018, 3:50 PM
Aklapper edited projects, added SRE-swift-storage; removed MediaWiki-General.Apr 28 2018, 4:15 PM

Confirming. A bit similar to T188885: Thumbnail missing for PDF file (when in PDF/A format?) and less to T151202: 429 Error generating thumbnail

• Vvjjkkii renamed this task from Particular image doesn't load (429 No Reason Phrase) to 50daaaaaaa.Jul 1 2018, 1:13 AM
• Vvjjkkii triaged this task as High priority.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
1339861mzb renamed this task from 50daaaaaaa to Particular image doesn't load (429 No Reason Phrase) .Jul 1 2018, 3:41 PM
1339861mzb updated the task description. (Show Details)
1339861mzb updated the task description. (Show Details)
JJMC89 raised the priority of this task from High to Needs Triage.Jul 1 2018, 4:33 PM
JJMC89 removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
JJMC89 added a subscriber: Aklapper.
matej_suchanek added a comment.Oct 8 2018, 4:40 PM

Another case:

Quiddity subscribed.Sep 23 2019, 3:20 AM
ovasileva subscribed.Sep 23 2019, 4:03 PM
AntiCompositeNumber added a project: Thumbor.May 15 2020, 2:28 AM
AntiCompositeNumber subscribed.May 15 2020, 4:22 AM

I'm not sure what's going on with https://commons.wikimedia.org/wiki/File:Sm%C4%9Bdavsk%C3%A1_hornatina,_Sloupsk%C3%BD_potok,_vodop%C3%A1d_01.jpg. ImageMagick processes locally it without error, so someone with production log access will have to look closer.

https://commons.wikimedia.org/wiki/File:R%C3%A9partition_Lemonia_dumi.svg is a different problem:

$ rsvg-convert -w 512 -f png -u -o Répartition_Lemonia_dumi.svg.png Répartition_Lemonia_dumi.svg
The SVG Répartition_Lemonia_dumi.svg has no dimensions

Running it though https://commons.wikimedia.org/wiki/Commons:Commons_SVG_Checker yields a more descriptive error message:

ERROR in <image> with id=rŽpartition Lemonia dumi.svg: Image has xlink:href attribute with data: element. Various configurations will not work (and may be blocked) by the Wikimedia software.

This image is a fake SVG consisting only of an embedded raster image. I've extracted the raster image and uploaded it to https://commons.wikimedia.org/wiki/File:R%C3%A9partition_Lemonia_dumi.png

matej_suchanek added a comment.May 21 2020, 8:49 AM

Thank you even for that one file!

AntiCompositeNumber renamed this task from Particular image doesn't load (429 No Reason Phrase) to HTTP 429 for File:Smědavská hornatina, Sloupský potok, vodopád 01.jpg at all thumbnail sizes.May 22 2020, 12:43 AM
AntiCompositeNumber added a comment.May 22 2020, 12:49 AM

What's interesting about this one is that it's always a 429 error. Even when thumbnail generation (presumably) has not been attempted for longer than the failure throttle timeout, I still get a 429 when requesting a thumbnail at any size.

HTTP/2 429 No Reason Phrase
date: Fri, 22 May 2020 00:46:00 GMT
server: Varnish
x-cache: cp1076 int
x-cache-status: int-front
server-timing: cache;desc="int-front"
strict-transport-security: max-age=106384710; includeSubDomains; preload
x-client-ip: ***
content-type: text/html; charset=utf-8
content-length: 1848
X-Firefox-Spdy: h2

https://upload.wikimedia.org/wikipedia/commons/thumb/2/20/Smědavská_hornatina,_Sloupský_potok,_vodopád_01.jpg/490px-Smědavská_hornatina,_Sloupský_potok,_vodopád_01.jpg

AntiCompositeNumber moved this task from Backlog to Broken or missing thumbnails on the Thumbor board.May 22 2020, 4:04 AM
AntiCompositeNumber claimed this task.May 30 2020, 4:22 PM
AntiCompositeNumber triaged this task as Medium priority.

Strike that, finally got a 500.

Request from *** via cp1076 frontend, Varnish XID 780802066
Upstream caches: cp1076 int
Error: 500, Internal Server Error at Sat, 30 May 2020 04:52:58 GMT

I wasn't able to reproduce the issue previously because it's in exiv2, not ImageMagick. Testing locally, I get

2020-05-30 05:00:51 thumbor:ERROR ERROR: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 140, in get_image
    self.context.request.image_url
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1024, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 609, in _fetch
    self.context.request.engine.load(fetch_result.buffer, extension)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 119, in load
    self.lcl[enginename].load(buffer, extension)
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/__init__.py", line 167, in load
    image_or_frames = self.create_image(buffer)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 77, in create_image
    self.read_exif(temp_file)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 146, in read_exif
    self.exif['Pyexiv2Orientation'] = metadata.get('Exif.Image.Orientation').value
  File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 190, in _get_value
    self._compute_value()
  File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 185, in _compute_value
    self._value = self._convert_to_python(self._raw_value)
  File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 311, in _convert_to_python
    raise ExifValueError(value, self.type)
ExifValueError: Invalid value for EXIF type [Short]: []

According to exiftool, there are a bunch of EXIF metadata keys with no values that shouldn't be there:

$ exiftool -verbose Smědavská_hornatina,_Sloupský_potok,_vodopád_01.jpg.1 

  ExifToolVersion = 11.85
  FileName = Sm..davsk.._hornatina,_Sloupsk.._potok,_vodop..d_01.jpg.1
  Directory = .
  FileSize = 1255502
  FileModifyDate = 1380903325
  FileAccessDate = 1590815002
  FileInodeChangeDate = 1590814915
  FilePermissions = 33188
  FileType = JPEG
  FileTypeExtension = JPG
  MIMEType = image/jpeg
JPEG APP1 (4030 bytes):
  ExifByteOrder = II
  + [IFD0 directory with 16 entries]
  | 0)  ImageDescription = 
  | Warning = Duplicate tag 0x010e ImageDescription in IFD0
  | 1)  ImageDescription = OLYMPUS DIGITAL CAMERA         
  | 2)  Make = OLYMPUS IMAGING CORP.  
  | 3)  Model = SP510UZ
  | 4)  Orientation = 
  | Warning = Duplicate tag 0x0112 Orientation in IFD0
  | 5)  Orientation = 1
  | 6)  XResolution = 
  | Warning = Duplicate tag 0x011a XResolution in IFD0
  | 7)  XResolution = 72 (72/1)
  | 8)  YResolution = 72 (72/1)
  | 9)  ResolutionUnit = 2
  | 10) Software = gthumb 2.11.3
  | 11) ModifyDate = 2010:06:06 17:43:25
  | 12) YCbCrPositioning = 2
  | 13) ExifOffset (SubDirectory) -->
  | + [ExifIFD directory with 33 entries]
  | | 0)  ExposureTime = 0.0125 (10/800)
  | | 1)  FNumber = 5.6 (56/10)
  | | 2)  ExposureProgram = 2
  | | 3)  ISO = 50
  | | 4)  ExifVersion = 0221
  | | 5)  DateTimeOriginal = 2010:06:06 17:43:25
  | | 6)  CreateDate = 2010:06:06 17:43:25
  | | 7)  ComponentsConfiguration = 1 2 3 0
  | | 8)  CompressedBitsPerPixel = 2 (2/1)
  | | 9)  ExposureCompensation = 0 (0/10)
  | | 10) MaxApertureValue = 2.97 (297/100)
  | | 11) MeteringMode = 5
  | | 12) LightSource = 0
  | | 13) Flash = 16
  | | 14) FocalLength = 6.3 (63/10)
  | | 15) MakerNoteOlympus (SubDirectory) -->
  | | + [MakerNotes directory with 52 entries]
  | | | 0)  SpecialMode = 0 0 0
  | | | 1)  Quality = 1
  | | | 2)  Macro = 0
  | | | 3)  BWMode = 6
  | | | 4)  DigitalZoom = 1 (100/100)
  | | | 5)  FocalPlaneDiagonal = 7.162 (7162/1000)
  | | | 6)  LensDistortionParams = -213 -407 -458 -184 -264 -230
  | | | 7)  CameraType = NORMAL  
  | | | 8)  TextInfo (SubDirectory) -->
  | | | + [pictureInfo directory]
  | | | | Resolution = 21
  | | | + [Camera Info directory]
  | | | | CameraType = D4322
  | | | 9)  CameraID = OLYMPUS DIGITAL CAMERA
  | | | 10) Olympus_0x020a = 0 0
  | | | 11) Olympus_0x0225 = 01
  | | | 12) PreCaptureFrames = 0
  | | | 13) WhiteBoard = 0
  | | | 14) OneTouchWB = 0
  | | | 15) WhiteBalanceBracket = 0
  | | | 16) WhiteBalanceBias = 0
  | | | 17) Olympus_0x0306 = 0
  | | | 18) Olympus_0x0400 = 1
  | | | 19) Olympus_0x0401 = 2
  | | | 20) Olympus_0x0402 = 16797705
  | | | 21) SceneMode = 1
  | | | 22) SerialNumber = 000J93C61030
  | | | 23) Firmware = 78
  | | | 24) DataDump = 00.a.1..+L....)..?,|d.289&. .&.(.L3..= 2.d..(.(J.....J..........[snip]
  | | | 25) CameraSettings (SubDirectory) -->
  | | | + [CameraSettings directory with 4 entries]
  | | | | 0)  CameraSettingsVersion = 0100
  | | | | 1)  PreviewImageValid = 1
  | | | | 2)  PreviewImageStart = 1274462
  | | | | 3)  PreviewImageLength = 69417
  | | | 26) SpecialMode = 
  | | | 27) Quality = 
  | | | 28) Macro = 
  | | | 29) BWMode = 
  | | | 30) DigitalZoom = 
  | | | 31) FocalPlaneDiagonal = 
  | | | 32) LensDistortionParams = 
  | | | 33) CameraType = 
  | | | 34) TextInfo (SubDirectory) -->
  | | | 35) CameraID = 
  | | | 36) Olympus_0x020a = 
  | | | 37) Olympus_0x0225 = 
  | | | 38) PreCaptureFrames = 
  | | | 39) WhiteBoard = 
  | | | 40) OneTouchWB = 
  | | | 41) WhiteBalanceBracket = 
  | | | 42) WhiteBalanceBias = 
  | | | 43) Olympus_0x0306 = 
  | | | 44) Olympus_0x0400 = 
  | | | 45) Olympus_0x0401 = 
  | | | 46) Olympus_0x0402 = 
  | | | 47) SceneMode = 
  | | | 48) SerialNumber = 
  | | | 49) Firmware = 
  | | | 50) DataDump = 
  | | | 51) CameraSettingsIFD (SubDirectory) -->
  | | 16) UserComment =                                                                [snip]
  | | 17) FlashpixVersion = 0100
  | | 18) ColorSpace = 1
  | | 19) ExifImageWidth = 2304
  | | 20) ExifImageHeight = 3072
  | | 21) InteropOffset (SubDirectory) -->
  | | + [InteropIFD directory with 2 entries]
  | | | 0)  InteropIndex = R98
  | | | 1)  InteropVersion = 0100
  | | 22) FileSource = 3
  | | 23) SceneType = 1
  | | 24) CustomRendered = 0
  | | 25) ExposureMode = 0
  | | 26) WhiteBalance = 0
  | | 27) DigitalZoomRatio = 0 (0/100)
  | | 28) SceneCaptureType = 0
  | | 29) GainControl = 0
  | | 30) Contrast = 0
  | | 31) Saturation = 0
  | | 32) Sharpness = 0
  | 14) PrintIM (SubDirectory) -->
  | Warning = [minor] Empty PrintIM data
  | Warning = Duplicate tag 0xc4a5 PrintIM in IFD0
  | 15) PrintIM (SubDirectory) -->
  | + [PrintIM directory with 37 entries]
  | | PrintIMVersion = 0300
  | | 0)  PrintIM_0x0001 = 1310740
  | | 1)  PrintIM_0x0002 = 1
  | | 2)  PrintIM_0x0003 = 238
  | | 3)  PrintIM_0x0007 = 0
  | | 4)  PrintIM_0x0008 = 0
  | | 5)  PrintIM_0x0009 = 0
  | | 6)  PrintIM_0x000a = 0
  | | 7)  PrintIM_0x000b = 310
  | | 8)  PrintIM_0x000c = 0
  | | 9)  PrintIM_0x000d = 0
  | | 10) PrintIM_0x000e = 334
  | | 11) PrintIM_0x0010 = 350
  | | 12) PrintIM_0x0020 = 594
  | | 13) PrintIM_0x0100 = 3
  | | 14) PrintIM_0x0101 = 255
  | | 15) PrintIM_0x0102 = 131
  | | 16) PrintIM_0x0103 = 131
  | | 17) PrintIM_0x0104 = 131
  | | 18) PrintIM_0x0105 = 131
  | | 19) PrintIM_0x0106 = 131
  | | 20) PrintIM_0x0107 = 8421504
  | | 21) PrintIM_0x0110 = 128
  | | 22) PrintIM_0x0200 = 0
  | | 23) PrintIM_0x0207 = 0
  | | 24) PrintIM_0x0208 = 0
  | | 25) PrintIM_0x0209 = 0
  | | 26) PrintIM_0x020a = 0
  | | 27) PrintIM_0x020b = 630
  | | 28) PrintIM_0x020d = 0
  | | 29) PrintIM_0x0220 = 654
  | | 30) PrintIM_0x0300 = 3
  | | 31) PrintIM_0x0301 = 255
  | | 32) PrintIM_0x0302 = 131
  | | 33) PrintIM_0x0303 = 131
  | | 34) PrintIM_0x0306 = 131
  | | 35) PrintIM_0x0310 = 128
  | | 36) PrintIM_0x0400 = 0
JPEG DQT (65 bytes):
JPEG DQT (65 bytes):
JPEG SOF0 (15 bytes):
  ImageWidth = 2304
  ImageHeight = 3072
  EncodingProcess = 0
  BitsPerSample = 8
  ColorComponents = 3
  YCbCrSubSampling = 1 2
JPEG DHT (29 bytes):
JPEG DHT (179 bytes):
JPEG DHT (29 bytes):
JPEG DHT (179 bytes):
JPEG SOS
  Warning = [minor] Error reading PreviewImage from file

In any case, not being able to figure out if we need to rotate a thumbnail is not a good reason to fail thumbnails for an entire file.

gerritbot added a comment.May 30 2020, 6:40 PM

Change 600400 had a related patch set uploaded (by AntiCompositeNumber; owner: AntiCompositeNumber):
[operations/software/thumbor-plugins@master] engine.imagemagick: Catch pyexiv2.ExifValueError also

https://gerrit.wikimedia.org/r/600400

gerritbot added a project: Patch-For-Review.May 30 2020, 6:40 PM
AntiCompositeNumber moved this task from Broken or missing thumbnails to Pending review on the Thumbor board.May 30 2020, 6:41 PM
gerritbot added a comment.Jun 3 2020, 10:48 AM

Change 600400 merged by Gilles:
[operations/software/thumbor-plugins@master] engine.imagemagick: Catch pyexiv2.ExifValueError also

https://gerrit.wikimedia.org/r/600400

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2020, 11:11 AM
AntiCompositeNumber moved this task from Pending review to To be packaged on the Thumbor board.Jun 3 2020, 8:14 PM
AntiCompositeNumber moved this task from Incoming to Thumbnail and file renderings on the Commons board.
• Gilles added a parent task: T254845: Build python-thumbor-wikimedia 2.9 Debian package and upload to apt.wikimedia.org.Jun 9 2020, 7:24 AM
• Gilles mentioned this in rTHMBREXT8a0c9f6010d6: Version bump.Jun 9 2020, 7:28 AM
gerritbot added a comment.Jun 9 2020, 7:31 AM

Change 603876 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.9

https://gerrit.wikimedia.org/r/603876

gerritbot added a project: Patch-For-Review.Jun 9 2020, 7:31 AM
gerritbot added a comment.Dec 10 2020, 9:00 AM

Change 603876 merged by Effie Mouzeli:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.9

https://gerrit.wikimedia.org/r/603876

Maintenance_bot removed a project: Patch-For-Review.Dec 10 2020, 9:11 AM
matej_suchanek closed this task as Resolved.Apr 15 2021, 9:49 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL