Page MenuHomePhabricator

thumbor error spawning ghostscript 'libcgroup initialization failed: Cgroup is not mounted'
Closed, ResolvedPublic

Description

spotted while observing thumbor logs

Sep  7 15:19:55 thumbor1001 thumbor@8817[98004]: ClientException: Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-hu-local-thumb.b9/thumbor/b/b9/Supnat5x22.JPG/250px-Supnat5x22.JPG 404 Not Found  [first 60 chars of res
ponse] <html><h1>Not Found</h1><p>The resource could not be found.<
Sep  7 15:19:55 thumbor1001 thumbor@8817[98004]: thumbor:WARNING ERROR retrieving image http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-hu-local-public.b9/b/b9/Supnat5x22.JPG: HTTP 404: Not Found
Sep  7 15:19:55 thumbor1001 thumbor@8817[98004]: tornado.access:WARNING 404 GET /wikipedia/hu/thumb/b/b9/Supnat5x22.JPG/250px-Supnat5x22.JPG (127.0.0.1) 73.75ms
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: swiftclient:ERROR Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikiversity-en-local-thumb/thumbor/0/02/The_Cerebellum.pdf/page15-593px-The_Cerebellum.pdf.jpg 404 Not Found  [fi
rst 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: Traceback (most recent call last):
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: service_token=self.service_token, **kwargs)
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1116, in get_object
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: http_response_content=body)
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: ClientException: Object GET failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikiversity-en-local-thumb/thumbor/0/02/The_Cerebellum.pdf/page15-593px-The_Cerebellum.pdf.jpg 404 Not Found  [fir
st 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: thumbor:ERROR ERROR: Traceback (most recent call last):
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 111, in get_image
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: self.context.request.image_url
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: value = future.result()
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: raise_exc_info(self._exc_info)
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 879, in run
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: yielded = self.gen.send(value)
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 525, in _fetch
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: self.context.request.engine.load(fetch_result.buffer, extension)
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 111, in load
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: self.lcl[enginename].load(buffer, extension)
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/thumbor/engines/__init__.py", line 162, in load
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: image_or_frames = self.create_image(buffer)
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/ghostscript/ghostscript.py", line 33, in create_image
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: png = self.get_png_for_page(buffer, page)
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/ghostscript/ghostscript.py", line 66, in get_png_for_page
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: return self.command(command)
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/__init__.py", line 101, in command
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: returncode
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: CommandError: (['/usr/bin/gs', '-sDEVICE=png16m', '-sOutputFile=%stdout', '-dFirstPage=15', '-dLastPage=15', '-r150', '-dBATCH', '-dNOPAUSE', '-dSAFER', '-q', '-f/tmp/tmpodcSeN/source_file']
, '', 'libcgroup initialization failed: Cgroup is not mounted\n', 81)
Sep  7 15:19:58 thumbor1001 thumbor@8813[98117]: thumbor:ERROR [BaseHandler] get_image failed for url `http%3A//ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikiversity-en-local-public/0/02/The_Cerebellum.pdf`. error: `(['/usr/bin/gs', '-sDEVICE=png16
m', '-sOutputFile=%stdout', '-dFirstPage=15', '-dLastPage=15', '-r150', '-dBATCH', '-dNOPAUSE', '-dSAFER', '-q', '-f/tmp/tmpodcSeN/source_file'], '', 'libcgroup initialization failed: Cgroup is not mounted\n', 81)`

Event Timeline

Currently thumbor production puppet has this configuration:

SUBPROCESS_CGEXEC_PATH = '/usr/bin/cgexec'
SUBPROCESS_CGROUP = 'memory:thumbor'

Meaning that all commands ran by thumbor are wrapped in a cgexec call. I would assume that the error is accurate and that we need to do something to make sure the thumbor cgroup's virtual disk is mounted?

How have you configured the cgroup for the production puppet?

Change 309267 had a related patch set uploaded (by Filippo Giunchedi):
thumbor: enable memory cgroup

https://gerrit.wikimedia.org/r/309267

Change 309267 merged by Filippo Giunchedi:
thumbor: enable memory cgroup

https://gerrit.wikimedia.org/r/309267

Mentioned in SAL [2016-09-08T09:46:18Z] <godog> roll-reboot thumbor machines to apply memory cgroup enablement T144938

Change 309279 had a related patch set uploaded (by Filippo Giunchedi):
thumbor: use native systemd memory limiting

https://gerrit.wikimedia.org/r/309279

Change 309279 merged by Filippo Giunchedi:
thumbor: use native systemd memory limiting

https://gerrit.wikimedia.org/r/309279

fgiunchedi claimed this task.

with native systemd memory cgroup this is no longer an issue, resolving