Page MenuHomePhabricator

Figure out Thumbor limits
Closed, ResolvedPublic

Description

File size, requested dimensions, memory, etc. We should seek sane values close to the old limits.

Event Timeline

Gilles created this task.Sep 7 2015, 8:14 PM
Gilles claimed this task.
Gilles raised the priority of this task from to Medium.
Gilles updated the task description. (Show Details)
Gilles added a project: Performance-Team.
Gilles added subscribers: Gilles, Aklapper.

Just for fun, let's try to attach the PNG bomb I'm going to run tests with here:

At the mediawiki level, a built-in error kicks in given the file's dimensions. But that wouldn't stop querying thumbor via varnish, it just prevents mediawiki from pointing to it.

Thumbor attempts to resize that image, and ends up crashing...

Interestingly, it outputs that warning on the server side:

/srv/thumbor/local/lib/python2.7/site-packages/PIL/Image.py:2261: DecompressionBombWarning: Image size (50625000000 pixels) exceeds limit of 89478485 pixels, could be decompression bomb DOS attack.
  DecompressionBombWarning)

But this is coming from PIL, not thumbor itself.

Thumbor doesn't support a maximum megapixels value, but does support maximum width and height. I've tried those out, but they didn't stop the PNG bomb. I think it's because the documentation for those limits is false advertisement: they only cap the destination width/height, and never actually look at the source dimensions.

I've filed two bugs with thumbor about this:

https://github.com/thumbor/thumbor/issues/604
https://github.com/thumbor/thumbor/issues/605

I might try writing support for a MP limit in thumbor and submitting it as a pull request.

It turns out that in production we don't apply the limit to JPG, as the sampling technique we use with IM means that we can resize even giant JPGs efficiently. I will have to verify whether PIL is capable of that or not.

And for PNGs, we use VIPS and don't set a limit. In practice on beta commons I was unable to get a thumbnail generated for the PNG bomb, I suspect that some limit from limit.sh kicked in. I'm going to make VIPS work on VM to verify that assumption.

I see two possible outcomes here in order to support giant JPGs. Either extending my patch for Thumbor by allowing to set a pixels limit per format (if PIL can handle those images efficiently), or implementing a VIPS engine for Thumbor.

Gilles closed this task as Resolved.Nov 26 2015, 4:10 PM

I think we should be generally fine in that area now, I just have to implement a solution like T117090 to avoid the disk filling up when using file storage.

Other than that giant JPGs and PNGs should be handled correctly, thanks to T119184 and T119185