Page MenuHomePhabricator

SVG files larger than 10 MB cannot be thumbnailed
Closed, ResolvedPublic

Description

https://commons.wikimedia.org/wiki/File:Thousand_Island,_St._Lawrence_River.svg

The thumbnails error out with HTTP 500 (or 429 if the thumbnail scaler throttling has been activated) with the message

Error creating thumbnail: Error reading SVG:Error domain 1 code 1 on line 3409 column 7 of file:///tmp/svg_09ebfac0817ad425c214af8a/localcopy_4c26893f84e0-1.svg: internal error: Huge input lookup

Related Objects

StatusAssignedTask
OpenNone
ResolvedMoritzMuehlenhoff
ResolvedMoritzMuehlenhoff
ResolvedKrenair
Resolved AlexMonk-WMF
Resolvedfgiunchedi
Resolved AlexMonk-WMF
ResolvedKrenair
Resolvedfgiunchedi
ResolvedKrenair
DeclinedNone
Resolvedmobrovac
ResolvedKrinkle
ResolvedKartikMistry
ResolvedKartikMistry
Resolvedbd808
InvalidNone
DeclinedNone
Resolved dduvall
Resolved dduvall

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Josve05a raised the priority of this task from to Needs Triage.Sep 8 2015, 4:06 PM
Josve05a updated the task description. (Show Details)
Josve05a added a subscriber: Josve05a.
Restricted Application added subscribers: Steinsplitter, Matanya, Aklapper. · View Herald TranscriptSep 8 2015, 4:06 PM
Josve05a renamed this task from SVG file is generating 429-error to SVG file is generating 429 and 500 errors.Sep 8 2015, 4:08 PM
Josve05a set Security to None.
Steinsplitter moved this task from Incoming to Backlog on the Commons board.Sep 8 2015, 5:19 PM
Tgr added a subscriber: Tgr.Sep 8 2015, 11:13 PM
Error creating thumbnail: Error reading SVG:Error domain 1 code 1 on line 3409 column 7 of file:///tmp/svg_09ebfac0817ad425c214af8a/localcopy_4c26893f84e0-1.svg: internal error: Huge input lookup

The SVG file is over 60 MB.

Jdforrester-WMF renamed this task from SVG file is generating 429 and 500 errors to SVG file is generating 429 and 500 errors because it is too large.Sep 10 2015, 1:15 AM
Jdforrester-WMF moved this task from Untriaged to Backlog on the Multimedia board.

Locally I can render the file (rsvg 2.26.3), but it probably depends on what version of libxml its linked against (google suggests error related to the XML_PARSE_HUGE constant [or lack thereof])

Note, its probably not because the image itself is too big, but because the embedded png images are too big (As an aside, the embedded png images also don't appear to be compressed properly, at all. If one extracted the embedded images, ran them through something like pngcrush, and then re-embedded them, it would probably reduce the file size significantly)

Looking through various things on the internet, I suspect that this would affect images that have embedded pngs (or other attributes) larger than 9.5 mb (Have not tested this theory)

Note, its probably not because the image itself is too big, but because the embedded png images are too big (As an aside, the embedded png images also don't appear to be compressed properly, at all. If one extracted the embedded images, ran them through something like pngcrush, and then re-embedded them, it would probably reduce the file size significantly)
Looking through various things on the internet, I suspect that this would affect images that have embedded pngs (or other attributes) larger than 9.5 mb (Have not tested this theory)

Looking around commons, seems to affect most images bigger than 10 mb. e.g. https://upload.wikimedia.org/wikipedia/commons/thumb/6/64/3D_image_of_Antihydrogen.svg/139px-3D_image_of_Antihydrogen.svg.png https://upload.wikimedia.org/wikipedia/commons/thumb/d/d1/2012_French_presidential_election_-_First_round_-_Majority_vote_%28Mayenne%29.svg/510px-2012_French_presidential_election_-_First_round_-_Majority_vote_%28Mayenne%29.svg.png

There are about 9007 images bigger then this, which are probably affected


I just read the rsvg docs. Looks like all we need to do is add the --unlimited flag to the command line (Assuming we have the right version of rsvg). Someone want to try and see if that works with whatever version is on the image scalers?

Tgr added a comment.Sep 11 2015, 11:21 PM
tgr@mw1153:~$ curl -s 'https://upload.wikimedia.org/wikipedia/commons/e/e7/Thousand_Island%2C_St._Lawrence_River.svg' > T111815_test.svg
tgr@mw1153:~$ /usr/bin/rsvg-convert -w 320 -h 200 -o T111815_test.png T111815_test.svg 
Error reading SVG:Error domain 1 code 1 on line 3409 column 7 of file:///home/tgr/T111815_test.svg: internal error: Huge input lookup


tgr@mw1153:~$ /usr/bin/rsvg-convert -w 320 -h 200 --unlimited -o T111815_test.png T111815_test.svg 
Unknown option --unlimited

Probably not the right version then.

Tgr added a comment.Sep 11 2015, 11:23 PM
tgr@mw1153:~$ /usr/bin/rsvg-convert --version
rsvg-convert version 2.40.2

Digging into the rsvg git repo, looks like upstream bug is https://bugzilla.gnome.org/show_bug.cgi?id=710310 and that the -u flag was added in 2.40.4 (commit aa1f447e2), so we just missed it :(

RP88 added a subscriber: RP88.Sep 12 2015, 10:48 AM

Yes, --unlimited works in newer versions so this should be fixed once we distro-upgrade (or backport?) librsvg:

$:andre\> /usr/bin/rsvg-convert --version
rsvg-convert version 2.40.10
$:andre\> curl -s 'https://upload.wikimedia.org/wikipedia/commons/e/e7/Thousand_Island%2C_St._Lawrence_River.svg' > T111815_test.svg
$:andre\> ls -l T111815_test.svg
-rw-rw-r--. 1 andre andre 67350922 Sep 12 15:32 T111815_test.svg
$:andre\> /usr/bin/rsvg-convert -w 320 -h 200 -o T111815_test.png T111815_test.svg
Error reading SVG:Error domain 1 code 5 on line 1 column 1 of file:///home/andre/T111815_test.svg: Extra content at the end of the document
$:andre\> /usr/bin/rsvg-convert -w 320 -h 200 --unlimited -o T111815_test.png T111815_test.svg
$:andre\>
Tgr renamed this task from SVG file is generating 429 and 500 errors because it is too large to SVG files larger than 10 MB cannot be thumbnailed.Sep 13 2015, 12:46 AM
Tgr updated the task description. (Show Details)
Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 15 2015, 6:13 AM
Aklapper triaged this task as Normal priority.Mar 8 2016, 10:53 PM
Amitie_10g added a subscriber: Amitie_10g.EditedApr 10 2016, 1:44 AM

Meanwhile the bug is resolved (or scheduled for resolving), could be a good idea to add {{HugeSVG}} at the Description in the Upload Wizard.

That bug is fixed on the new jessie image scaler using 2.4.16 (tested locally, it's not yet pooled into the set of active scalers in production). The original file from this bug was removed for copyright reasons, but I confirmed it with https://commons.wikimedia.org/wiki/File:3D_image_of_Antihydrogen.svg

Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptJun 22 2016, 11:19 AM
Amitie_10g added a comment.EditedJun 22 2016, 4:44 PM

I see the files that {{HugeSVG}} ins transcluded, but them're still unable to render as PNG.

https://commons.wikimedia.org/wiki/Special:WhatLinksHere/Template:HugeSVG

I'll watch this thread and check these files periodically.

The update is not deployed yet, Moritz is working on it. Please subscribe to the blocking task T112421: Update rsvg on the image scalers to 2.40.16 (to solve several SVG rendering issues) for updates.

Tgr added a comment.Jul 4 2016, 7:16 PM

I can still reproduce this, despite the scalers running 2.40.16.

I can still reproduce this, despite the scalers running 2.40.16.

Has the --unlimited flag been added?

The librsvg bug mentions the flag has security implications, maybe it should be reviewed by Security?

3D_image_of_Antihydrogen.svg was in fact misleading, while the new librsvg offers a method to whilelist limits in libxml using RSVG_HANDLE_FLAG_UNLIMITED, this is not enabled in our conversion process so far. As for the security implications: They certainly exist ( by someone uploading malformed SVG files which triggers "billion laughs attack"-style resource consumption), but they are mititigated by the cgroups resource limits and execution timeout we impose on the launched librsvg processes. I'm not sure how exhaustive the list from https://commons.wikimedia.org/wiki/Special:WhatLinksHere/Template:HugeSVG is, but if the number is generally low, one possible approach would be to pass the RSVG_HANDLE_FLAG_UNLIMITED option for SVG files whitelisted by administrators.

dpatrick added a subscriber: dpatrick.EditedJul 7 2016, 9:55 PM

We discussed this issue in the Security Team meeting. Our consensus is that it is okay to add --unlimited given that we have mitigations in place, as @MoritzMuehlenhoff mentioned above.

Additionally, Bawolff has added a unit test to ensure that Billion Laughs-style attacks in SVG files are detected and blocked appropriately.

We discussed this issue in the Security Team meeting. Our consensus is that it is okay to add --unlimited given that we have mitigations in place, as @MoritzMuehlenhoff mentioned above.

Just as a further note, I wanted to mention we also check SVG files for entity related DOS at time of upload.

Sounds good to me. I'll run some tests with "--unlimited" next week and if all if fine, I'll enable it on the image scalers.

Sounds good to me. I'll run some tests with "--unlimited" next week and if all if fine, I'll enable it on the image scalers.

Did you find time for this?

not yet, but should be able to have a look at this end of the week or next week

I ran a bot and got a (huge) list of files SVGs > 10 MB (more than 8000 ones). I'll post the list soon, so, I don't have the time to see every file for rendering errors, so, please take it.

Change 303548 had a related patch set uploaded (by Muehlenhoff):
Support scaling of huge SVGs

https://gerrit.wikimedia.org/r/303548

Johan added a subscriber: Johan.Aug 11 2016, 8:57 AM

Since this has been tagged for inclusion in Tech News: When will this fix go into production? Do you know?

@Johan This still needs more tests before it can be enabled in production, I'll update this Phab task when that has happened.

Johan added a comment.Aug 11 2016, 9:03 AM

OK, thanks. (:

Change 303548 merged by Muehlenhoff:
Support scaling of huge SVGs

https://gerrit.wikimedia.org/r/303548

Mentioned in SAL [2016-08-18T08:59:34Z] <moritzm> enabled scaling of huge SVGs on image scalers (T111815)

This is now enabled on the image scalers. If anyone still runs into a huge SVG which can't be thumbnailed, please post the filename for further analysis.

@Johan : This is now live, but maybe wait another 24 hours before announcing it. I checked the logs and before this patch was merged we had 76 failed huge SVG thumbnailings over the course of a day. So I should be able to tell you tomorrow whether there's still open bugs wrt scaling huge SVGs.

MoritzMuehlenhoff closed this task as Resolved.Aug 19 2016, 7:16 AM

Seems all fine, previously on average 250 SVG thumbailings failed daily due to size limitations, but since the patch was merged that has stopped completely. @Johan Should be ok to User-Notice now.

Johan added a comment.Aug 19 2016, 7:17 PM

OK, thanks. It'll go out in the newsletter that's distributed on Monday.

This comment was removed by Josve05a.
This comment was removed by matmarex.