There might be a delay between Nginx shipping a request to Thumbor and Thumbor picking it up, which could explain Nginx timeouts. Recording the request time from Nginx's POV should allow us to figure that out.
Description
Details
Revisions and Commits
rTHMBREXT Thumbor Plugins | |||
Restricted Differential Revision | rTHMBREXTb8d01d672299 Include request date header given by nginx in the response |
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | • Gilles | T121388 Service-based thumbnailing re-architecture in production with Thumbor | |||
Resolved | • Gilles | T150746 Nginx timeouts on Thumbor | |||
Resolved | • Gilles | T161535 Track nginx request time in Thumbor debug headers |
Event Timeline
Change 344968 had a related patch set uploaded (by Gilles):
[operations/puppet@production] Set nginx request time as a header passed to Thumbor
Change 344968 merged by Filippo Giunchedi:
[operations/puppet@production] Set nginx request time as a header passed to Thumbor
Change 345283 had a related patch set uploaded (by Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 0.1.37
Change 345283 merged by Filippo Giunchedi:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 0.1.37
Doesn't seem to work right:
gilles@thumbor1001:~$ curl -I http://localhost:8809/wikipedia/en/thumb/9/9f/WUVI.jpg/720px-WUVI.jpg HTTP/1.1 200 OK Engine: wikimedia_thumbor.engine.imagemagick Content-Length: 76102 Processing-Utime: 412 Content-Disposition: inline;filename*=UTF-8''WUVI.jpg Wikimedia-Original-Container: wikipedia-en-local-public.9f Request-Date: None Thumbor-Parameters: {"width": "720", "image": "http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.9f/9/9f/WUVI.jpg", "filters": "conditional_sharpen(0.0,0.8,1.0,0.0,0.85):format(jpg)"} Server: Thumbor/6.0.1 Wikimedia-Thumbnail-Container: wikipedia-en-local-thumb.9f Etag: "d4ead8672c3091844afb4b162c4cf72d6a0af3a0" Processing-Time: 453 Convert-Time: 72 Date: Thu, 30 Mar 2017 16:21:35 GMT Xkey: File:WUVI.jpg Wikimedia-Original-Path: 9/9f/WUVI.jpg Content-Type: image/jpeg Exiftool-Time: 109 Exiftool-Time: 128 Exiftool-Time: 139 Wikimedia-Thumbnail-Path: thumbor/9/9f/WUVI.jpg/720px-WUVI.jpg
Hah, it's been working the entire time. Of course it doesn't work if you hit Thumbor directly, only if you hit nginx:
gilles@thumbor1001:~$ curl -I http://localhost:8800/wikipedia/en/thumb/9/9f/WUVI.jpg/720px-WUVI.jpg HTTP/1.1 200 OK Server: nginx/1.11.13 Date: Mon, 25 Sep 2017 11:02:31 GMT Content-Type: image/jpeg Content-Length: 76102 Connection: close Swift-Time: 83 Content-Disposition: inline;filename*=UTF-8''WUVI.jpg Wikimedia-Original-Container: wikipedia-en-local-public.9f Request-Date: Monday, 25-Sep-2017 11:02:31 GMT Thumbor-Parameters: {"width": "720", "image": "https://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-en-local-public.9f/9/9f/WUVI.jpg", "filters": "conditional_sharpen(0.0,0.8,1.0,0.0,0.85):format(jpg)"} Wikimedia-Thumbnail-Container: wikipedia-en-local-thumb.9f Etag: "d4ead8672c3091844afb4b162c4cf72d6a0af3a0" Xkey: File:WUVI.jpg Wikimedia-Original-Path: 9/9f/WUVI.jpg Wikimedia-Thumbnail-Path: 9/9f/WUVI.jpg/720px-WUVI.jpg X-Upstream: 127.0.0.1:8837
Change 380480 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/vagrant@master] Pass request time from nginx to thumbor
Change 380480 merged by jenkins-bot:
[mediawiki/vagrant@master] Pass request time from nginx to thumbor
Change 380483 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Thumbor: expose Nginx request time
Change 380483 merged by Filippo Giunchedi:
[operations/puppet@production] Thumbor: expose Nginx request time
Mentioned in SAL (#wikimedia-operations) [2017-09-26T08:09:53Z] <godog> roll-restart swift-proxy to apply https://gerrit.wikimedia.org/r/#/c/380483/ - T161535