VIPS doesn't scale PNGs larger than roughly 140 megapixels
Closed, ResolvedPublic

Description

At some point, things just become insane, but we should probably see which resource is the limiting factor for "huge" (>140MP) files.

Example files:
[[:file:Agriculture_characterisation_map,_Flanders.png]], [[:file:Basemap_for_Afil.png]], [[:file:Drapeau_hillion.png]]


Version: unspecified
Severity: normal

bzimport added a subscriber: Unknown Object (MLST).
bzimport set Reference to bz52050.
Bawolff created this task.Via LegacyJul 25 2013, 7:24 PM
Bawolff added a comment.Via ConduitAug 27 2013, 7:22 PM

Testing locally (Important, locally I'm not using cgroups, along with probably a different version of vips (vips-7.34.2), so these local tests might not mean much)

[[:file:Agriculture_characterisation_map,_Flanders.png]],

This one worked fine locally (It did require more than 100mb of memory, but still less than 400mb).

[[:file:Basemap_for_Afil.png]]

Seems to need a bigger temporary file (575 mb seemed to do it)

, [[:file:Drapeau_hillion.png]]

Seems to need a 738 mb temp file.


So it would appear increasing $wgMaxShellFileSize to say 1 GB would decrease the number of images we fail to resize by quite a bit. Not sure if that's something viable to do. At the very least, probably not until bug 52203 is fixed.

Nemo_bis added a comment.Via ConduitSep 12 2013, 2:12 PM

Doesn't block bug 9497 as that's closed.

Nemo_bis added a comment.Via ConduitSep 12 2013, 2:16 PM

(In reply to comment #1)

At the very least, probably not until bug 52203 is
fixed.

Well, if your bug 52203 comment 4 is correct, raising the limit to fix this bug would be a workaround for that bug too, because if uninterrupted the extension cleans up its garbage by itself.
The existing workaround now in place seems to make disk storage not a problem, but I'm sure that's not the only consideration.

Bawolff added a comment.Via ConduitSep 12 2013, 3:29 PM

(In reply to comment #3)

(In reply to comment #1)
> At the very least, probably not until bug 52203 is
> fixed.

Well, if your bug 52203 comment 4 is correct, raising the limit to fix this
bug
would be a workaround for that bug too, because if uninterrupted the
extension
cleans up its garbage by itself.
The existing workaround now in place seems to make disk storage not a
problem,
but I'm sure that's not the only consideration.

J had a tweak to the vips command line that should significantly reduce size of temp files (which should be deployed next week). That may stop many of the failures.

Proper solution to the temp file issue would maybe be to create temporary working directory for vips and delete everything in directory when done.

Bawolff added a comment.Via ConduitSep 12 2013, 9:12 PM

In regards to j's tweak - http://test.wikipedia.org/wiki/File:Huge.png is 160 mp and scaling fine now. True test will come when changes hit commons next week.

Bawolff added a comment.Via ConduitSep 21 2013, 3:35 PM

Ok, these are the biggest pngs (all pngs bigger than 170 MP) on commons:

MariaDB [commonswiki_p]> select img_name, img_width*img_height from image where img_media_type = 'BITMAP' and img_minor_mime = 'png' and img_major_mime = 'image' and img_width *img_height > 170000000 order by img_width*img_height desc;
+-----------------------------------------------------------+----------------------+

img_nameimg_width*img_height

+-----------------------------------------------------------+----------------------+

Iowa_County,_Wisconsin,_Courthouse_-_Icon_-_3000-dpi.png4161055734
Mandelbrot_fractal_large.png715776516
Mandelbrot_High_Resolution.png600000000
Iowa_County,_Wisconsin,_Courthouse_-_Icon_-1000-dpi.png462349287
Gonders_flats.png375955914
U-undStraßenbahnGleisplanBerlinundUmgebung2013.png293840000
Union_Stock_Yards,_Omaha,_1899.png260564824
Union_Stock_Yards,_Omaha,_1925.png259465592
Union_Stock_Yards,_Omaha,_1958.png258749920
Union_Stock_Yards,_Omaha,_1916.png258264072
Union_Stock_Yards,_Omaha,_1999.png257947776
Union_Stock_Yards,_Omaha,_1887.png257896800
Redstone_Rocket_Test_Stand_HAER_AL-129-A_sheet_1_of_7.png241231872
World4-isometric_north.png200950784
Rensselaerswyck_Original_Map_Large.png195843032
Drapeau_hillion.png192853712
Income_inequality_in_the_United_States,_1979-2007.png174420000

+-----------------------------------------------------------+----------------------+

enwiki has no pngs bigger than 170 MP

I tested the top 5, they all seem to render. I will note, that [[commons:file:Iowa_County,_Wisconsin,_Courthouse_-_Icon_-_3000-dpi.png]] doesn't seem to render the courthouse part of the image (compare with [[commons:file:Iowa_County,_Wisconsin,_Courthouse_-_Icon_-1000-dpi.png]]). I'm not sure if that's a problem with our rendering, or if the original image is just broken. I'm going to go with the original image is just broken unless someone comes up with evidence to the contrary.

In any case, marking this bug fixed

MarkTraceur added a project: Multimedia.Via WebDec 3 2014, 6:25 PM
Gilles raised the priority of this task from "Normal" to "Unbreak Now!".Via WebDec 4 2014, 10:12 AM
Gilles moved this task to Closed on the Multimedia workboard.
hashar removed a subscriber: hashar.Via WebDec 4 2014, 10:22 AM
Gilles lowered the priority of this task from "Unbreak Now!" to "Normal".Via ConduitDec 4 2014, 11:20 AM

Add Comment