SVG and VIPS engines expect temp folder to stay a while when it doesn't
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Gilles
	Oct 17 2016, 7:39 AM

Description

Having long-lived temp folders isn't a reasonable expectation anymore:

Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: 2016-10-17 07:04:44,645 8837 thumbor:ERROR [BaseHandler] get_image failed for url `http%3A//ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-commons-local-public.89/8/89/Flag_of_Antigua_and_Barbuda.svg`. error: `[Errno 2] No such file or directory: '/srv/thumbor/tmp/thumbor@8837/tmp_nWyEZ/source_file'`

Oct 17 07:03:02 thumbor1001 thumbor@8825[71257]: CommandError: (['/usr/bin/vips', 'shrink', '/srv/thumbor/tmp/thumbor@8825/tmpv2KVW9', '/srv/thumbor/tmp/thumbor@8825/tmp7Cl9Rr/vips_result.png', '31', '31'], '', 'vips__file_open_write: unable to open file "/srv/thumbor/tmp/thumbor@8825/tmp7Cl9Rr/vips_result.png" for writing\n', 1)

The solution is simply to stop creating a temp folder and named temp files and just used regular temp files.

Details

	Subject	Repo	Branch	Lines +/-
	Upgrade to 0.1.28	operations/debs/python-thumbor-wikimedia	master	+74 -28

Customize query in gerrit

Revisions and Commits

rTHMBREXT Thumbor Plugins
	rTHMBREXTb916c63cfbdf Avoid long-lived temp folders

Related Objects
Search...

Status	Assigned	Task
Resolved	• Gilles	T121388 Service-based thumbnailing re-architecture in production with Thumbor
Resolved	• Gilles	T139606 add thumbor to production infrastructure
Resolved	• Gilles	T148361 SVG and VIPS engines expect temp folder to stay a while when it doesn't

Event Timeline

• Gilles created this task.Oct 17 2016, 7:39 AM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptOct 17 2016, 7:39 AM

Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: 2016-10-17 07:04:44,645 8837 thumbor:ERROR ERROR: Traceback (most recent call last):
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 111, in get_image
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: self.context.request.image_url
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: value = future.result()
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: raise_exc_info(self._exc_info)
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 879, in run
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: yielded = self.gen.send(value)
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 525, in _fetch
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: self.context.request.engine.load(fetch_result.buffer, extension)
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 118, in load
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: self.lcl[enginename].load(buffer, extension)
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: File "/usr/lib/python2.7/dist-packages/thumbor/engines/__init__.py", line 162, in load
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: image_or_frames = self.create_image(buffer)
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/svg/svg.py", line 34, in create_image
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: self.prepare_source(buffer)
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/__init__.py", line 85, in prepare_source
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: with open(self.source, 'w') as source:
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: IOError: [Errno 2] No such file or directory: '/srv/thumbor/tmp/thumbor@8837/tmp_nWyEZ/source_file'
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: 2016-10-17 07:04:44,645 8837 thumbor:ERROR [BaseHandler] get_image failed for url `http%3A//ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-commons-local-public.89/8/89/Flag_of_Antigua_and_Barbuda.svg`. error: `[Errno 2] No such file or directory: '/srv/thumbor/tmp/thumbor@8837/tmp_nWyEZ/source_file'`
Oct 17 07:04:44 thumbor1001 thumbor@8837[74679]: 2016-10-17 07:04:44,646 8837 tornado.access:ERROR 500 GET /wikipedia/commons/thumb/8/89/Flag_of_Antigua_and_Barbuda.svg/535px-Flag_of_Antigua_and_Barbuda.svg.png (127.0.0.1) 219.06ms

This looks like the temp folder we create is getting nuked because it's too old or something like that. It might be that the code expects the folder to keep existing. I.e. we should stop using named temporary files like that and just use regular temporary files. There's no need to reuse the folder and the name like that.

This one looks like a different issue:

Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: 2016-10-17 07:06:47,721 8825 thumbor:ERROR ERROR: Traceback (most recent call last):
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 111, in get_image
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: self.context.request.image_url
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: value = future.result()
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: raise_exc_info(self._exc_info)
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 879, in run
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: yielded = self.gen.send(value)
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 525, in _fetch
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: self.context.request.engine.load(fetch_result.buffer, extension)
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 118, in load
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: self.lcl[enginename].load(buffer, extension)
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/thumbor/engines/__init__.py", line 162, in load
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: image_or_frames = self.create_image(buffer)
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/svg/svg.py", line 55, in create_image
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: return super(Engine, self).create_image(png)
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 123, in create_image
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: im.read(blob=buffer)
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/wand/image.py", line 2042, in read
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: self.raise_exception()
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: File "/usr/lib/python2.7/dist-packages/wand/resource.py", line 218, in raise_exception
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: raise e
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: BlobError: zero-length blob not permitted `' @ error/blob.c/BlobToImage/349
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: 2016-10-17 07:06:47,721 8825 thumbor:ERROR [BaseHandler] get_image failed for url `http%3A//ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-commons-local-public.03/0/03/Go-shichi_no_kiri_crest_2.svg`. error: `zero-length blob not permitted `' @ error/blob.c/BlobToImage/349`
Oct 17 07:06:47 thumbor1001 thumbor@8825[71257]: 2016-10-17 07:06:47,722 8825 tornado.access:ERROR 500 GET /wikipedia/commons/thumb/0/03/Go-shichi_no_kiri_crest_2.svg/200000px-Go-shichi_no_kiri_crest_2.svg.png (127.0.0.1) 160.47ms

A temporary fetch problem from Swift, perhaps? Re-requesting thumbnails on the same original doesn't trigger errors on thumbor1001.

The other ones are the same missing source_file issue.

• Gilles renamed this task from Investigate failing SVGs to SVG engine: IOError: [Errno 2] No such file or directory: '/srv/thumbor/tmp/thumbor@8811/tmp04oo68/source_file'.Oct 17 2016, 8:02 AM

• Gilles mentioned this in T148362: Investigate failing PNGs.Oct 17 2016, 8:05 AM

• Gilles merged a task: T148362: Investigate failing PNGs.

• Gilles renamed this task from SVG engine: IOError: [Errno 2] No such file or directory: '/srv/thumbor/tmp/thumbor@8811/tmp04oo68/source_file' to SVG and VIPS engines expect temp folder to stay a while when it doesn't.Oct 17 2016, 8:08 AM

• Gilles updated the task description. (Show Details)

• Gilles merged a task: T148360: Investigate failing TIFFs.Oct 17 2016, 8:11 AM

• Gilles added a commit: rTHMBREXTb916c63cfbdf: Avoid long-lived temp folders.Oct 17 2016, 9:02 AM

Change 316305 had a related patch set uploaded (by Gilles):
Upgrade to 0.1.28

https://gerrit.wikimedia.org/r/316305

gerritbot added a project: Patch-For-Review.Oct 17 2016, 9:44 AM

• Gilles moved this task from Backlog to Pending review on the Thumbor board.Oct 17 2016, 9:52 AM

• Gilles moved this task from Inbox, needs triage to Blocked (old) on the Performance-Team board.

• Gilles moved this task from Blocked (old) to Doing (old) on the Performance-Team board.

Change 316305 merged by Filippo Giunchedi:
Upgrade to 0.1.28