We should be using HTTP_LOADER_CURL_ASYNC_HTTP_CLIENT = True
Description
Description
Revisions and Commits
Revisions and Commits
| rTHMBREXT Thumbor Plugins | |||
| rTHMBREXT9c658b06bd3a Work around Tornado default 100MB remote file limit | |||
| rTHMBREXT99a781ca831d Work around Tornado default 100MB remote file limit | |||
| rMWVA mediawiki/vagrant | |||
| rMWVAb072a101d106 Enable Tornado async curl loader for Thumbor | |||
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Gilles | T121388 Service-based thumbnailing re-architecture in production with Thumbor | |||
| Resolved | • Gilles | T139606 add thumbor to production infrastructure | |||
| Resolved | • Gilles | T145768 Thumbor can't load source files bigger than 100MB |
Event Timeline
Comment Actions
Turning HTTP_LOADER_CURL_ASYNC_HTTP_CLIENT on resulted in catastrophic failure after a short time:, with a thumbor instance flooding the logs with the following:
Sep 15 11:39:56 thumbor1001 thumbor@8814[64240]: File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 865, in start Sep 15 11:39:56 thumbor1001 thumbor@8814[64240]: fd_obj, handler_func = self._handlers[fd] Sep 15 11:39:56 thumbor1001 thumbor@8814[64240]: KeyError: 33 Sep 15 11:39:56 thumbor1001 thumbor@8814[64240]: tornado.application:ERROR Exception in callback None Sep 15 11:39:56 thumbor1001 thumbor@8814[64240]: Traceback (most recent call last): Sep 15 11:39:56 thumbor1001 thumbor@8814[64240]: File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 865, in start Sep 15 11:39:56 thumbor1001 thumbor@8814[64240]: fd_obj, handler_func = self._handlers[fd] Sep 15 11:39:56 thumbor1001 thumbor@8814[64240]: KeyError: 33 Sep 15 11:39:59 thumbor1001 thumbor@8814[64240]: tornado.application:ERROR Exception in callback None Sep 15 11:39:59 thumbor1001 thumbor@8814[64240]: Traceback (most recent call last): Sep 15 11:39:59 thumbor1001 thumbor@8814[64240]: File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 865, in start Sep 15 11:39:59 thumbor1001 thumbor@8814[64240]: fd_obj, handler_func = self._handlers[fd] Sep 15 11:39:59 thumbor1001 thumbor@8814[64240]: KeyError: 33 Sep 15 11:39:59 thumbor1001 thumbor@8814[64240]: tornado.application:ERROR Exception in callback None Sep 15 11:39:59 thumbor1001 thumbor@8814[64240]: Traceback (most recent call last): Sep 15 11:39:59 thumbor1001 thumbor@8814[64240]: File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 865, in start
Comment Actions
Actually, I see now that it's a single instance stuck in that state. Maybe not related to that config change I'm trying? Although it wasn't happening before...
Comment Actions
I'm restarting the thumbor instances and waiting to see if that crazy error reappears.
Comment Actions
So far so good regarding the crazy error, it's not happening anymore. Maybe it had to do with restarting the thumbor instances?
Comment Actions
Trying to fix that mysterious epic failure is probably a fool's errand, as it lies in the bowels of Tornado.
I think that a better idea could be to write a custom loader that downloads the source file to a temporary file, populates the result's buffer variable with the temp file name. Or maybe just the beginning of the actual buffer for mime detection to work, and find a way to pass the temporary file name differently for the engines to consume.
Then all the engines we have should be able to consume a temporary file as input. In fact most of them now dump the buffer into a temporary file, so we would actually save time.
Even the imagemagick engine should be ok dealing with a temporary file instead of a buffer.