Page MenuHomePhabricator
Create Task
Maniphest T139606

add thumbor to production infrastructure
Closed, ResolvedPublic
Actions

Description

tracking here what needs to happen on the production infrastructure to happen for thumbor:

  • hardware provisioning
  • LVS setup
  • modify rewrite.py to duplicate a fraction of thumbnail requests to thumbor (some of it already in mediawiki-vagrant) T139484
  • thumbor puppetization (includes module, role, etc)
  • grant thumbor-admins access https://gerrit.wikimedia.org/r/#/c/302471/
  • provision swift account for thumbor and provide write access to relevant containers
  • monitoring and alarming
  • setup firejail for thumbor, similar to imagescaler

HW provisioning

thumbor is expected to be purely CPU/network bound. With ideally less, but at least similar, requirements as the current image scalers. We can order dedicated hardware and in the mean time re-use some of recently decommissioned hardware.

rewrite.py integration

Implementation is part of [[https://phabricator.wikimedia.org/rMWVA9971189bbc715e7df2d7f19a5ad5edb6ca4c2399 | mediawiki-vagrant ]] and should be ported to production and its configuration puppetized: T139484

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• GillesT121388 Service-based thumbnailing re-architecture in production with Thumbor
 Resolved• GillesT139606 add thumbor to production infrastructure
 Resolved• GillesT139484 Dual-serve a portion of the production traffic to Thumbor
 Resolved• GillesT140072 Bring changes to rewrite.py done on Vagrant to production puppet
 ResolvedNoneT144928 thumbor/exiftool deadlock, likely full pipe
 ResolvedfgiunchediT144938 thumbor error spawning ghostscript 'libcgroup initialization failed: Cgroup is not mounted'
 Resolved• GillesT144956 thumbor handling of originals 404
 ResolvedfgiunchediT145051 Unsupported header value None
 ResolvedfgiunchediT145061 invalid literal for int() with base 10:
 Resolved• GillesT145377 SVG type check too slow in production
 Resolved• GillesT145505 'NoneType' object has no attribute 'lstrip'
 Declined• GillesT145610 rsvg exit status -11
 Resolved• GillesT145612 Extremely noisy ffmpeg errors
 Resolved• GillesT145614 0px thumbnail requests should fail more elegantly
 Resolved• GillesT145616 djvu failure for very high page number
 Resolved• GillesT145617 pdf failure
 Resolved• GillesT145618 Thumbor SVG regexp insufficient
 Declined• GillesT145623 thumbor memory limits for main process and subprocesses
 Resolved• GillesT145626 thumbor ffmpeg pipe deadlock
 Resolved• GillesT145632 Separate 404s into their own log
 Resolved• GillesT145637 Use intermediary high-quality JPEGs rather than PNGs for PDF thumbnailing
 Resolved• GillesT145638 VIPS engine should generate JPG when dealing with TIFFs and not have the IM engine read it
 Resolved• GillesT145768 Thumbor can't load source files bigger than 100MB
 Resolved• GillesT145769 Archive file thumbs not working
 Invalid• GillesT145784 Report more metrics with statsd
 Resolved• GillesT145785 Make the 100MB+ test files downloaded from their source instead of being in the git repo
 Resolved• GillesT145878 thumbor imagemagick filling up /tmp on thumbor1002
 Resolved• GillesT150208 Avoid thumbor generating log files > 1GB
 Resolved• GillesT146143 Figure out a way to live-debug running production thumbor processes
 Resolved• GillesT146262 Temp files not cleaned up on conversion error
 Resolved• GillesT147412 Thumbor times out on large files sometimes
 Resolved• GillesT147417 thumbor: Some video files not recognized
 Resolved• GillesT147918 Log thumbnail requests that fail on Thumbor and not on Mediawiki and vice versa
 Resolved• GillesT147919 Thumbor fails on gifs where Mediawiki doesn't
 Resolved• GillesT147921 Extracted ICC profile don't get cleaned up
 Duplicate• GillesT148360 Investigate failing TIFFs
 Resolved• GillesT148361 SVG and VIPS engines expect temp folder to stay a while when it doesn't
 Duplicate• GillesT148362 Investigate failing PNGs
 Resolved• GillesT148363 Missing original for video 500s instead of 404ing
 Resolved• GillesT148410 Investigate source of thumbnail 302 redirects
 Resolved• GillesT149981 Ask firejail upstream about ability to turn off pid namespacing
 Resolved• GillesT149985 Set IM Thumbor engine environment variables
 Resolved• GillesT150641 Investigate differences in status codes between thumbor and image scalers
 Resolved• GillesT150743 Investigate SVG default language behavior on non-English wikis for Thumbor
 Resolved• GillesT150745 Reimplement various rate-limiting mechanisms in Thumbor
 Duplicate• GillesT150751 Thumbor/rewrite.py should check for redirects regardless of status code of the response
 Duplicate• GillesT150753 Thumbor 504s on images Mediawiki 404s on
 Declined• GillesT150755 Mediawiki 301s on some images that Thumbor renders succesfully
 Resolved• GillesT150756 Thumbor 504s on a few images Mediawiki 500s on
 Declined• GillesT150761 Thumbor 503s on an image Mediawiki renders successfully
 Resolved• GillesT150642 Match cache headers between thumbor and mediawiki
 Resolved• GillesT151072 Improve Content-Disposition support in Thumbor
 Resolved• GillesT150757 Fix memory leaks in Thumbor plugins
 Resolved• GillesT150749 Thumbor should reject some malformed thumbnail URLs
 Resolved• GillesT150758 Thumbor should know about the lossless thumbnail parameter
 Resolved• GillesT150760 Thumbor 404s when the original has a ? in its filename
 Resolved• GillesT151453 Thumbor errors when %0A is in the filename part of the request
 Resolved• GillesT151454 Thumbor hangs on some TIFF files
 Resolved• GillesT151455 Thumbor errors on some GIF files
 Resolved• GillesT151565 Upgrade gifsicle to 1.88-2~bpo8+1 on Thumbor boxes
 Resolved• GillesT151459 Nginx time limit should be a bit higher than Thumbor subprocess time limit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Gilles closed subtask T145612: Extremely noisy ffmpeg errors as Resolved.Oct 5 2016, 11:51 AM
• Gilles closed subtask T145614: 0px thumbnail requests should fail more elegantly as Resolved.
• Gilles created subtask T147412: Thumbor times out on large files sometimes.Oct 5 2016, 11:57 AM
• Gilles closed subtask T145616: djvu failure for very high page number as Resolved.
• Gilles closed subtask T145618: Thumbor SVG regexp insufficient as Resolved.
• Gilles closed subtask T145768: Thumbor can't load source files bigger than 100MB as Resolved.Oct 5 2016, 12:01 PM
• Gilles closed subtask T145617: pdf failure as Resolved.Oct 5 2016, 12:13 PM
• Gilles closed subtask T145637: Use intermediary high-quality JPEGs rather than PNGs for PDF thumbnailing as Resolved.
• Gilles closed subtask T145769: Archive file thumbs not working as Resolved.Oct 5 2016, 12:29 PM
• Gilles closed subtask T139484: Dual-serve a portion of the production traffic to Thumbor as Resolved.Oct 5 2016, 12:39 PM
• Gilles created subtask T147417: thumbor: Some video files not recognized.Oct 5 2016, 12:48 PM
• Gilles closed subtask T145638: VIPS engine should generate JPG when dealing with TIFFs and not have the IM engine read it as Resolved.Oct 5 2016, 12:54 PM
• Gilles closed subtask T145626: thumbor ffmpeg pipe deadlock as Resolved.Oct 6 2016, 1:23 PM
• Gilles closed subtask T147417: thumbor: Some video files not recognized as Resolved.Oct 11 2016, 8:20 AM
• Gilles created subtask T147918: Log thumbnail requests that fail on Thumbor and not on Mediawiki and vice versa.Oct 12 2016, 9:33 AM
• Gilles created subtask T147919: Thumbor fails on gifs where Mediawiki doesn't.Oct 12 2016, 9:53 AM
• Gilles created subtask T147921: Extracted ICC profile don't get cleaned up.Oct 12 2016, 11:32 AM
• Gilles closed subtask T146262: Temp files not cleaned up on conversion error as Resolved.Oct 12 2016, 11:38 AM
• Gilles closed subtask T146143: Figure out a way to live-debug running production thumbor processes as Resolved.Oct 12 2016, 11:41 AM
• Gilles closed subtask T145878: thumbor imagemagick filling up /tmp on thumbor1002 as Resolved.Oct 12 2016, 11:43 AM
• Gilles closed subtask T147412: Thumbor times out on large files sometimes as Resolved.Oct 13 2016, 5:38 AM
• Gilles added a comment.Oct 13 2016, 5:41 AM

"monitoring and alarming" is still unchecked on this task. Is that true? Is there still something to do there? I thought we had paging alarms raised before when disk usage filled up.

• Gilles removed a subtask: T145504: Gifsicle engine: AttributeError: 'Engine' object has no attribute 'exif'.Oct 13 2016, 5:45 AM
• Gilles moved this task from Backlog to Doing on the Thumbor board.Oct 13 2016, 5:54 AM
• Gilles closed subtask T147919: Thumbor fails on gifs where Mediawiki doesn't as Resolved.Oct 13 2016, 3:40 PM
• Gilles closed subtask T147921: Extracted ICC profile don't get cleaned up as Resolved.Oct 13 2016, 3:46 PM
• Gilles created subtask T148360: Investigate failing TIFFs.Oct 17 2016, 7:37 AM
• Gilles created subtask T148361: SVG and VIPS engines expect temp folder to stay a while when it doesn't.
• Gilles created subtask T148362: Investigate failing PNGs.Oct 17 2016, 7:40 AM
• Gilles created subtask T148363: Missing original for video 500s instead of 404ing.
• Gilles closed subtask T148363: Missing original for video 500s instead of 404ing as Resolved.Oct 17 2016, 11:23 AM
• Gilles created subtask T148410: Investigate source of thumbnail 302 redirects.Oct 17 2016, 11:44 AM
• Gilles closed subtask T148361: SVG and VIPS engines expect temp folder to stay a while when it doesn't as Resolved.
• Gilles closed subtask T145632: Separate 404s into their own log as Resolved.Oct 18 2016, 5:52 PM
Dzahn unsubscribed.Oct 18 2016, 5:56 PM
• Gilles closed subtask T145623: thumbor memory limits for main process and subprocesses as Declined.Oct 24 2016, 1:27 PM
• Gilles created subtask T148962: Record OOM kills as a metric with mtail.
fgiunchedi reopened subtask T145878: thumbor imagemagick filling up /tmp on thumbor1002 as Open.Oct 29 2016, 1:30 AM
• Gilles created subtask T149981: Ask firejail upstream about ability to turn off pid namespacing.Nov 4 2016, 8:22 AM
• Gilles created subtask T149985: Set IM Thumbor engine environment variables.Nov 4 2016, 9:13 AM
• Gilles closed subtask T149981: Ask firejail upstream about ability to turn off pid namespacing as Resolved.Nov 7 2016, 9:10 AM
• Gilles closed subtask T149985: Set IM Thumbor engine environment variables as Resolved.Nov 9 2016, 6:30 PM
• Gilles closed subtask T147918: Log thumbnail requests that fail on Thumbor and not on Mediawiki and vice versa as Resolved.Nov 14 2016, 9:30 AM
• Gilles created subtask T150641: Investigate differences in status codes between thumbor and image scalers.
• Gilles created subtask T150642: Match cache headers between thumbor and mediawiki.
• Gilles closed subtask T145878: thumbor imagemagick filling up /tmp on thumbor1002 as Resolved.Nov 14 2016, 9:35 AM
• Gilles created subtask T150643: Thumbor OOMs too much.Nov 14 2016, 9:40 AM
• Gilles removed a subtask: T150643: Thumbor OOMs too much.Nov 14 2016, 9:44 AM
• Gilles closed subtask T150641: Investigate differences in status codes between thumbor and image scalers as Resolved.Nov 15 2016, 12:29 PM
• Gilles created subtask T151072: Improve Content-Disposition support in Thumbor.Nov 18 2016, 5:23 PM
• Gilles removed a subtask: T148962: Record OOM kills as a metric with mtail.Nov 21 2016, 1:36 PM
• Gilles added a subtask: T151441: Thumbor should handle "temp" thumbnail requests.Nov 23 2016, 10:28 AM
• Gilles added a subtask: T150757: Fix memory leaks in Thumbor plugins.Nov 23 2016, 12:08 PM
• Gilles added a subtask: T151067: Implement rate limiter in Thumbor.
• Gilles added a subtask: T151066: Implement PoolCounter support in Thumbor.
• Gilles added a subtask: T151065: Implement DC-local cache failure limiter in Thumbor.
• Gilles added a subtask: T150749: Thumbor should reject some malformed thumbnail URLs.
• Gilles added a subtask: T150754: Thumbor can't render a few SVGs that Mediawiki can.
• Gilles added a subtask: T150758: Thumbor should know about the lossless thumbnail parameter.
• Gilles added a subtask: T150760: Thumbor 404s when the original has a ? in its filename.
• Gilles added a subtask: T150741: Thumbor should reject thumbnail requests that are the same size as the original or bigger.Nov 23 2016, 12:10 PM
• Gilles created subtask T151453: Thumbor errors when %0A is in the filename part of the request.Nov 23 2016, 1:16 PM
• Gilles removed a project: Patch-For-Review.
• Gilles created subtask T151454: Thumbor hangs on some TIFF files.Nov 23 2016, 1:54 PM
• Gilles created subtask T151455: Thumbor errors on some GIF files.Nov 23 2016, 1:58 PM
• Gilles created subtask T151456: Thumbor original file download limit should be 4GB.Nov 23 2016, 2:26 PM
• Gilles removed a subtask: T151456: Thumbor original file download limit should be 4GB.
• Gilles created subtask T151459: Nginx time limit should be a bit higher than Thumbor subprocess time limit.Nov 23 2016, 2:44 PM
elukey unsubscribed.Nov 24 2016, 11:30 AM
• Gilles removed fgiunchedi as the assignee of this task.Nov 24 2016, 1:42 PM
• Gilles created subtask T151553: Add request URL to thumbor errors.
• Gilles created subtask T151554: Track incoming HTTP request count on the Thumbor boxes.Nov 24 2016, 1:46 PM
• Gilles moved this task from Doing to Backlog on the Thumbor board.Nov 24 2016, 4:00 PM
• Gilles created subtask T151851: Thumbor resource consumption is spiky.Nov 29 2016, 8:48 AM
• Gilles closed subtask T148410: Investigate source of thumbnail 302 redirects as Resolved.Dec 1 2016, 12:45 PM
• Gilles closed subtask T151459: Nginx time limit should be a bit higher than Thumbor subprocess time limit as Resolved.Dec 2 2016, 11:07 AM
• Gilles closed subtask T150642: Match cache headers between thumbor and mediawiki as Resolved.Dec 2 2016, 8:39 PM
• Gilles closed subtask T151072: Improve Content-Disposition support in Thumbor as Resolved.
• Gilles closed subtask T150757: Fix memory leaks in Thumbor plugins as Resolved.Dec 2 2016, 9:25 PM
• Gilles closed subtask T150758: Thumbor should know about the lossless thumbnail parameter as Resolved.Dec 2 2016, 9:31 PM
• Gilles closed subtask T151454: Thumbor hangs on some TIFF files as Resolved.Dec 2 2016, 9:35 PM
• Gilles closed subtask T151455: Thumbor errors on some GIF files as Resolved.Dec 2 2016, 9:39 PM
• Gilles closed subtask T151453: Thumbor errors when %0A is in the filename part of the request as Resolved.Dec 2 2016, 9:42 PM
• Gilles closed subtask T150760: Thumbor 404s when the original has a ? in its filename as Resolved.
• Gilles closed subtask T150749: Thumbor should reject some malformed thumbnail URLs as Resolved.Dec 2 2016, 9:45 PM
• Gilles moved this task from Backlog to Doing on the Thumbor board.Dec 6 2016, 2:25 PM
• Gilles added a comment.Dec 6 2016, 2:29 PM

I'm going to move all the remaining tasks to the parent, since for all intents and purposes, Thumbor has been running in production for some time now, just not serving its results yet.

• Gilles closed this task as Resolved.Dec 6 2016, 2:30 PM
• Gilles claimed this task.
• Gilles removed a subtask: T144479: Ensure thumbor container access is preserved by mw filebackend setzoneaccess.
• Gilles removed a subtask: T151441: Thumbor should handle "temp" thumbnail requests.
• Gilles removed a subtask: T151067: Implement rate limiter in Thumbor.
• Gilles removed a subtask: T151066: Implement PoolCounter support in Thumbor.Dec 6 2016, 2:32 PM
• Gilles removed a subtask: T151065: Implement DC-local cache failure limiter in Thumbor.
• Gilles removed a subtask: T150754: Thumbor can't render a few SVGs that Mediawiki can.
• Gilles removed a subtask: T150741: Thumbor should reject thumbnail requests that are the same size as the original or bigger.
• Gilles removed a subtask: T151553: Add request URL to thumbor errors.
• Gilles removed a subtask: T151554: Track incoming HTTP request count on the Thumbor boxes.
• Gilles removed a subtask: T151851: Thumbor resource consumption is spiky.Dec 6 2016, 2:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits