Thumbor errors on some GIF files
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Gilles
	Nov 23 2016, 1:58 PM

Description

https://en.wikipedia.org/wiki/File:Rahansvirtues.gif

2016-11-23 13:56:01,283 8841 thumbor:DEBUG [ImagesHandler] translate: {'lang': None, 'end': u'Rahansvirtues', 'extension': u'gif', 'language': u'en', 'format': u'gif', 'shard2': u'ba', 'filename': u'Rahansvirtues', 'project': u'wikipedia', 'width': u'275', 'lossy': None, 'qlow': None, 'specialpath': None, 'seek': None, 'shard1': u'b', 'page': None}
2016-11-23 13:56:01,284 8841 thumbor:DEBUG [Swift] get: u'wikipedia-en-local-thumb.ba' u'thumbor/b/ba/Rahansvirtues.gif/275px-Rahansvirtues.gif'
2016-11-23 13:56:01,354 8841 thumbor:DEBUG [Swift] missing
2016-11-23 13:56:01,354 8841 thumbor:DEBUG [REQUEST_STORAGE] get: http%3A//ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.ba/b/ba/Rahansvirtues.gif
2016-11-23 13:56:01,355 8841 thumbor:DEBUG [REQUEST_STORAGE] missing
2016-11-23 13:56:01,355 8841 thumbor:DEBUG [HTTPS] load_sync: http%3A//ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.ba/b/ba/Rahansvirtues.gif
2016-11-23 13:56:01,550 8841 thumbor:DEBUG [HTTPS] return_contents: /tmp/tmp0DeIcE
2016-11-23 13:56:01,550 8841 thumbor:DEBUG [Proxy] Looking for a gif engine
2016-11-23 13:56:01,558 8841 PIL.Image:DEBUG Error closing: 'NoneType' object has no attribute 'close'
2016-11-23 13:56:01,558 8841 thumbor:DEBUG [REQUEST_STORAGE] put: http%3A//ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.ba/b/ba/Rahansvirtues.gif
2016-11-23 13:56:01,559 8841 thumbor:ERROR Ignored error handling exif for reorientation
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/__init__.py", line 231, in _get_exif_segment
    segment = ExifSegment(None, None, self.exif, 'ro')
AttributeError: 'Engine' object has no attribute 'exif'
2016-11-23 13:56:01,559 8841 thumbor:ERROR Ignored error handling exif for reorientation
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/__init__.py", line 231, in _get_exif_segment
    segment = ExifSegment(None, None, self.exif, 'ro')
AttributeError: 'Engine' object has no attribute 'exif'
2016-11-23 13:56:01,559 8841 thumbor:DEBUG Format specified: gif
2016-11-23 13:56:01,559 8841 thumbor:DEBUG Image format specified as .gif.
2016-11-23 13:56:01,559 8841 thumbor:DEBUG Content Type of image/gif detected.
2016-11-23 13:56:01,566 8841 PIL.Image:DEBUG Importing BmpImagePlugin
2016-11-23 13:56:01,566 8841 PIL.Image:DEBUG Importing BufrStubImagePlugin
2016-11-23 13:56:01,566 8841 PIL.Image:DEBUG Importing CurImagePlugin
2016-11-23 13:56:01,567 8841 PIL.Image:DEBUG Importing DcxImagePlugin
2016-11-23 13:56:01,567 8841 PIL.Image:DEBUG Importing DdsImagePlugin
2016-11-23 13:56:01,567 8841 PIL.Image:DEBUG Importing EpsImagePlugin
2016-11-23 13:56:01,568 8841 PIL.Image:DEBUG Importing FitsStubImagePlugin
2016-11-23 13:56:01,568 8841 PIL.Image:DEBUG Importing FliImagePlugin
2016-11-23 13:56:01,569 8841 PIL.Image:DEBUG Importing FpxImagePlugin
2016-11-23 13:56:01,570 8841 PIL.Image:DEBUG Importing FtexImagePlugin
2016-11-23 13:56:01,570 8841 PIL.Image:DEBUG Importing GbrImagePlugin
2016-11-23 13:56:01,570 8841 PIL.Image:DEBUG Importing GifImagePlugin
2016-11-23 13:56:01,570 8841 PIL.Image:DEBUG Importing GribStubImagePlugin
2016-11-23 13:56:01,570 8841 PIL.Image:DEBUG Importing Hdf5StubImagePlugin
2016-11-23 13:56:01,571 8841 PIL.Image:DEBUG Importing IcnsImagePlugin
2016-11-23 13:56:01,573 8841 PIL.Image:DEBUG Importing IcoImagePlugin
2016-11-23 13:56:01,573 8841 PIL.Image:DEBUG Importing ImImagePlugin
2016-11-23 13:56:01,574 8841 PIL.Image:DEBUG Importing ImtImagePlugin
2016-11-23 13:56:01,574 8841 PIL.Image:DEBUG Importing IptcImagePlugin
2016-11-23 13:56:01,575 8841 PIL.Image:DEBUG Importing JpegImagePlugin
2016-11-23 13:56:01,575 8841 PIL.Image:DEBUG Importing Jpeg2KImagePlugin
2016-11-23 13:56:01,575 8841 PIL.Image:DEBUG Importing McIdasImagePlugin
2016-11-23 13:56:01,575 8841 PIL.Image:DEBUG Importing MicImagePlugin
2016-11-23 13:56:01,575 8841 PIL.Image:DEBUG Importing MpegImagePlugin
2016-11-23 13:56:01,576 8841 PIL.Image:DEBUG Importing MpoImagePlugin
2016-11-23 13:56:01,576 8841 PIL.Image:DEBUG Importing MspImagePlugin
2016-11-23 13:56:01,576 8841 PIL.Image:DEBUG Importing PalmImagePlugin
2016-11-23 13:56:01,577 8841 PIL.Image:DEBUG Importing PcdImagePlugin
2016-11-23 13:56:01,577 8841 PIL.Image:DEBUG Importing PcxImagePlugin
2016-11-23 13:56:01,577 8841 PIL.Image:DEBUG Importing PdfImagePlugin
2016-11-23 13:56:01,577 8841 PIL.Image:DEBUG Importing PixarImagePlugin
2016-11-23 13:56:01,577 8841 PIL.Image:DEBUG Importing PngImagePlugin
2016-11-23 13:56:01,578 8841 PIL.Image:DEBUG Importing PpmImagePlugin
2016-11-23 13:56:01,578 8841 PIL.Image:DEBUG Importing PsdImagePlugin
2016-11-23 13:56:01,578 8841 PIL.Image:DEBUG Importing SgiImagePlugin
2016-11-23 13:56:01,578 8841 PIL.Image:DEBUG Importing SpiderImagePlugin
2016-11-23 13:56:01,578 8841 PIL.Image:DEBUG Importing SunImagePlugin
2016-11-23 13:56:01,579 8841 PIL.Image:DEBUG Importing TgaImagePlugin
2016-11-23 13:56:01,579 8841 PIL.Image:DEBUG Importing TiffImagePlugin
2016-11-23 13:56:01,579 8841 PIL.Image:DEBUG Importing WebPImagePlugin
2016-11-23 13:56:01,580 8841 PIL.Image:DEBUG Importing WmfImagePlugin
2016-11-23 13:56:01,580 8841 PIL.Image:DEBUG Importing XbmImagePlugin
2016-11-23 13:56:01,581 8841 PIL.Image:DEBUG Importing XpmImagePlugin
2016-11-23 13:56:01,581 8841 PIL.Image:DEBUG Importing XVThumbImagePlugin
2016-11-23 13:56:01,582 8841 thumbor:ERROR [GIF_ENGINE] invalid gif engine result for url `/wikipedia/en/thumb/b/ba/Rahansvirtues.gif/275px-Rahansvirtues.gif`.
2016-11-23 13:56:01,582 8841 thumbor:ERROR [GIF_ENGINE] invalid gif engine result for url `/wikipedia/en/thumb/b/ba/Rahansvirtues.gif/275px-Rahansvirtues.gif`.

Details

	Subject	Repo	Branch	Lines +/-
	Upgrade to 0.1.30	operations/debs/python-thumbor-wikimedia	master	+5 K -230

Customize query in gerrit

Revisions and Commits

rTHMBREXT Thumbor Plugins
	rTHMBREXT82e829ecf2d7 Add partially corrupt GIF file that fails with gifsicle < 1.88

Related Objects
Search...

Status	Assigned	Task
Resolved	• Gilles	T121388 Service-based thumbnailing re-architecture in production with Thumbor
Resolved	• Gilles	T139606 add thumbor to production infrastructure
Resolved	• Gilles	T151455 Thumbor errors on some GIF files
Resolved	• Gilles	T151565 Upgrade gifsicle to 1.88-2~bpo8+1 on Thumbor boxes

Event Timeline

• Gilles created this task.Nov 23 2016, 1:58 PM

Other example: https://commons.wikimedia.org/wiki/File:LOVE_A+T_LOVEigf.gif

This one's original starts off animated and blanks itself.

This error is probably related and showed up when stopping the Thumbor instance:

2016-11-23 14:29:53,444 8841 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 215, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 331, in finish_request
    callback=inner,
  File "/usr/lib/python2.7/dist-packages/thumbor/context.py", line 276, in queue
    self._execute_in_foreground(operation, callback)
  File "/usr/lib/python2.7/dist-packages/thumbor/context.py", line 263, in _execute_in_foreground
    result.set_result(operation())
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 267, in _load_results
    results = context.request.engine.read(image_extension, quality)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 132, in read
    ret = self.__getattr__('read')(extension, quality)
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/gif.py", line 107, in read
    with Image.open(BytesIO(self.buffer)) as image:
  File "/usr/lib/python2.7/dist-packages/PIL/Image.py", line 2319, in open
    % (filename if filename else fp))
IOError: cannot identify image file <_io.BytesIO object at 0x7ff27036c890>

This one: https://uk.wikipedia.org/wiki/%D0%A4%D0%B0%D0%B9%D0%BB:Jokie.gif

Fails for what seems to be a different reason:

2016-11-23 14:32:39,660 8841 thumbor:DEBUG [ImagesHandler] translate: {'lang': None, 'end': u'Jokie', 'extension': u'gif', 'language': u'uk', 'format': u'gif', 'shard2': u'ac', 'filename': u'Jokie', 'project': u'wikipedia', 'width': u'150', 'lossy': None, 'qlow': None, 'specialpath': None, 'seek': None, 'shard1': u'a', 'page': None}
2016-11-23 14:32:39,660 8841 thumbor:DEBUG [Swift] get: u'wikipedia-uk-local-thumb.ac' u'thumbor/a/ac/Jokie.gif/150px-Jokie.gif'
2016-11-23 14:32:39,705 8841 thumbor:DEBUG [Swift] missing
2016-11-23 14:32:39,706 8841 thumbor:DEBUG [REQUEST_STORAGE] get: http%3A//ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-uk-local-public.ac/a/ac/Jokie.gif
2016-11-23 14:32:39,706 8841 thumbor:DEBUG [REQUEST_STORAGE] missing
2016-11-23 14:32:39,706 8841 thumbor:DEBUG Importing: wikimedia_thumbor.loader.video
2016-11-23 14:32:39,707 8841 thumbor:DEBUG Importing: wikimedia_thumbor.loader.https
2016-11-23 14:32:39,709 8841 thumbor:DEBUG [HTTPS] load_sync: http%3A//ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-uk-local-public.ac/a/ac/Jokie.gif
2016-11-23 14:32:39,771 8841 thumbor:DEBUG [HTTPS] return_contents: /tmp/tmpPViLwI
2016-11-23 14:32:39,772 8841 thumbor:DEBUG [Proxy] Looking for a gif engine
2016-11-23 14:32:39,778 8841 PIL.Image:DEBUG Error closing: 'NoneType' object has no attribute 'close'
2016-11-23 14:32:39,779 8841 thumbor:DEBUG [REQUEST_STORAGE] put: http%3A//ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-uk-local-public.ac/a/ac/Jokie.gif
2016-11-23 14:32:39,779 8841 thumbor:ERROR Ignored error handling exif for reorientation
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/__init__.py", line 231, in _get_exif_segment
    segment = ExifSegment(None, None, self.exif, 'ro')
AttributeError: 'Engine' object has no attribute 'exif'
2016-11-23 14:32:39,779 8841 thumbor:ERROR Ignored error handling exif for reorientation
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/__init__.py", line 231, in _get_exif_segment
    segment = ExifSegment(None, None, self.exif, 'ro')
AttributeError: 'Engine' object has no attribute 'exif'
2016-11-23 14:32:39,788 8841 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 879, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 180, in get_image
    self.filters_runner.apply_filters(thumbor.filters.PHASE_AFTER_LOAD, transform)
  File "/usr/lib/python2.7/dist-packages/thumbor/filters/__init__.py", line 81, in apply_filters
    callback()
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 178, in transform
    self.context.transformer.transform(after_transform_cb)
  File "/usr/lib/python2.7/dist-packages/thumbor/transformer.py", line 102, in transform
    self.smart_detect()
  File "/usr/lib/python2.7/dist-packages/thumbor/transformer.py", line 137, in smart_detect
    self.do_image_operations()
  File "/usr/lib/python2.7/dist-packages/thumbor/transformer.py", line 230, in do_image_operations
    callback=inner
  File "/usr/lib/python2.7/dist-packages/thumbor/context.py", line 276, in queue
    self._execute_in_foreground(operation, callback)
  File "/usr/lib/python2.7/dist-packages/thumbor/context.py", line 263, in _execute_in_foreground
    result.set_result(operation())
  File "/usr/lib/python2.7/dist-packages/thumbor/transformer.py", line 213, in img_operation_worker
    self.auto_crop()
  File "/usr/lib/python2.7/dist-packages/thumbor/transformer.py", line 283, in auto_crop
    self.engine.crop(crop_left, crop_top, crop_right, crop_bottom)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 160, in crop
    return self.__getattr__('crop')(left, top, right, bottom)
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/gif.py", line 74, in crop
    self.update_image_info()
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/gif.py", line 41, in update_image_info
    self.image_size = size.groups()[0].split('x')
AttributeError: 'NoneType' object has no attribute 'groups'
2016-11-23 14:32:39,788 8841 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 879, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 180, in get_image
    self.filters_runner.apply_filters(thumbor.filters.PHASE_AFTER_LOAD, transform)
  File "/usr/lib/python2.7/dist-packages/thumbor/filters/__init__.py", line 81, in apply_filters
    callback()
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 178, in transform
    self.context.transformer.transform(after_transform_cb)
  File "/usr/lib/python2.7/dist-packages/thumbor/transformer.py", line 102, in transform
    self.smart_detect()
  File "/usr/lib/python2.7/dist-packages/thumbor/transformer.py", line 137, in smart_detect
    self.do_image_operations()
  File "/usr/lib/python2.7/dist-packages/thumbor/transformer.py", line 230, in do_image_operations
    callback=inner
  File "/usr/lib/python2.7/dist-packages/thumbor/context.py", line 276, in queue
    self._execute_in_foreground(operation, callback)
  File "/usr/lib/python2.7/dist-packages/thumbor/context.py", line 263, in _execute_in_foreground
    result.set_result(operation())
  File "/usr/lib/python2.7/dist-packages/thumbor/transformer.py", line 213, in img_operation_worker
    self.auto_crop()
  File "/usr/lib/python2.7/dist-packages/thumbor/transformer.py", line 283, in auto_crop
    self.engine.crop(crop_left, crop_top, crop_right, crop_bottom)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 160, in crop
    return self.__getattr__('crop')(left, top, right, bottom)
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/gif.py", line 74, in crop
    self.update_image_info()
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/gif.py", line 41, in update_image_info
    self.image_size = size.groups()[0].split('x')
AttributeError: 'NoneType' object has no attribute 'groups'

• Gilles moved this task from Inbox, needs triage to To-do: Goals prioritized current Quarter on the Performance-Team board.Nov 23 2016, 8:23 PM

elukey unsubscribed.Nov 24 2016, 11:31 AM

• Gilles created subtask T151565: Upgrade gifsicle to 1.88-2~bpo8+1 on Thumbor boxes.Nov 24 2016, 2:59 PM

• Gilles moved this task from To-do: Goals prioritized current Quarter to Doing (old) on the Performance-Team board.Nov 24 2016, 3:03 PM

The second example is actually the same issue manifesting itself, which should be resolved by the gifsicle upgrade.

• Gilles added a commit: rTHMBREXT82e829ecf2d7: Add partially corrupt GIF file that fails with gifsicle < 1.88.Nov 24 2016, 3:38 PM

• Gilles moved this task from Backlog to Doing on the Thumbor board.Nov 24 2016, 3:53 PM

• Gilles moved this task from Doing to To be packaged on the Thumbor board.Nov 28 2016, 10:48 AM

Change 324919 had a related patch set uploaded (by Gilles):
Upgrade to 0.1.30

https://gerrit.wikimedia.org/r/324919

gerritbot added a project: Patch-For-Review.Dec 2 2016, 3:11 PM

Change 324919 merged by Filippo Giunchedi:
Upgrade to 0.1.30

https://gerrit.wikimedia.org/r/324919

fgiunchedi closed subtask T151565: Upgrade gifsicle to 1.88-2~bpo8+1 on Thumbor boxes as Resolved.Dec 2 2016, 7:15 PM

Thumbor now doing what Mediawiki can't on those files:

Dec  2 21:38:24 ms-fe1001 proxy-server: HTTP status code mismatch. Mediawiki: 500 Thumbor: 200 URL: http://upload.wikimedia.org/wikipedia/en/thumb/b/ba/Rahansvirtues.gif/123px-Rahansvirtues.gif

Thumbor errors on some GIF filesClosed, ResolvedPublicActions