$wgMaxAnimatedGifArea is not honored by Thumbor
Closed, ResolvedPublic

Description

After proposing a file size restriction related to $wgMaxAnimatedGifArea limit, I noticed animation when generating new thumbnail sizes. Previously cached image do not animate. This happens even when both input (2,000 × 1,600 × 50 frames = 160 mp) and output (1,600 × 1,280 × 50 frames = 102 mp) exceed the 100 mp limit scaling (tested on Interfering waves simulation.gif).

Documentation for the sibling property $wgMaxImageArea suggests VipsScaler ignores these limits, but only for PNG and JPEG.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 18 2017, 4:34 PM
Bawolff added a subscriber: Bawolff.

I suspect this is due to thumbor.

Gilles added a subscriber: Gilles.Sep 8 2017, 7:08 AM

Indeed, I wasn't aware of that limit and it hasn't been ported over to Thumbor. Seems simple enough to reproduce.

Gilles triaged this task as Normal priority.Sep 8 2017, 7:08 AM
Gilles claimed this task.
Gilles added a project: Performance-Team.
Gilles renamed this task from $wgMaxAnimatedGifArea is not honored to $wgMaxAnimatedGifArea is not honored by Thumbor.

Change 376674 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 1.3

https://gerrit.wikimedia.org/r/376674

Change 376675 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/vagrant@master] Leverage new MAX_ANIMATED_GIF_AREA Thumbor option

https://gerrit.wikimedia.org/r/376675

Change 376676 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Thumbor: enable new MAX_ANIMATED_GIF_AREA option

https://gerrit.wikimedia.org/r/376676

Change 376674 merged by Filippo Giunchedi:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 1.3

https://gerrit.wikimedia.org/r/376674

Change 376676 merged by Filippo Giunchedi:
[operations/puppet@production] Thumbor: enable new MAX_ANIMATED_GIF_AREA option

https://gerrit.wikimedia.org/r/376676

Mentioned in SAL (#wikimedia-operations) [2017-09-11T09:40:20Z] <godog> roll-restart thumbor to apply changes for gif max animated area - T173580

I should have read the task description better, I missed the fact that it's the size multiplied by the amount of frames, not just the size of a single frame.

Change 377226 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 1.4

https://gerrit.wikimedia.org/r/377226

SQL requests to compile list of potentially affected files:

SELECT img_name, img_width * img_height * SUBSTRING(img_metadata, LOCATE('"frameCount";i:', img_metadata) + 15, LOCATE(';', SUBSTRING(img_metadata, LOCATE('"frameCount";i:', img_metadata) + 15)) - 1 ) as pixels FROM image WHERE img_minor_mime = 'gif' HAVING pixels > 100000000

SELECT DISTINCT oi_name FROM (SELECT oi_name, oi_width * oi_height * SUBSTRING(oi_metadata, LOCATE('"frameCount";i:', oi_metadata) + 15, LOCATE(';', SUBSTRING(oi_metadata, LOCATE('"frameCount";i:', oi_metadata) + 15)) - 1 ) as pixels FROM oldimage WHERE oi_minor_mime = 'gif' HAVING pixels > 100000000) overlimit

Checking manually for just Commons, where there are 151K GIFs, that's 468 potentially affected files in the image table and 121 files in the oldimage table (with a decent amount of overlap, I'm sure). Should be quick to purge once the list has been generated for every wiki.

Change 377226 merged by Filippo Giunchedi:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 1.4

https://gerrit.wikimedia.org/r/377226

Mentioned in SAL (#wikimedia-operations) [2017-09-11T14:40:06Z] <godog> roll-restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/377264/ and upgrade to 1.4 - T173580 T174997

File mentioned in the description fixed via purging.

Gilles closed this task as Resolved.Sep 11 2017, 7:32 PM

All potentially affected GIFs should have been purged now.

SELECT img_name, img_width * img_height * SUBSTRING(img_metadata, LOCATE('"frameCount";i:', img_metadata) + 15, LOCATE(';', SUBSTRING(img_metadata, LOCATE('"frameCount";i:', img_metadata) + 15)) - 1 ) as pixels FROM image WHERE img_minor_mime = 'gif' HAVING pixels > 100000000

I find SUBSTRING_INDEX to be slightly more readable and using the img_media_mime index cuts the runtime to under 5 minute.. See my GIF check report.

Change 376675 merged by jenkins-bot:
[mediawiki/vagrant@master] Leverage new MAX_ANIMATED_GIF_AREA Thumbor option

https://gerrit.wikimedia.org/r/376675