VIPS doesn't scale PNGs larger than roughly 140 megapixels
Closed, ResolvedPublic

Description

At some point, things just become insane, but we should probably see which resource is the limiting factor for "huge" (>140MP) files.

Example files:
[[:file:Agriculture_characterisation_map,_Flanders.png]], [[:file:Basemap_for_Afil.png]], [[:file:Drapeau_hillion.png]]


Version: unspecified
Severity: normal

bzimport set Reference to bz52050.
bzimport added a subscriber: Unknown Object (MLST).
Bawolff created this task.Jul 25 2013, 7:24 PM

Testing locally (Important, locally I'm not using cgroups, along with probably a different version of vips (vips-7.34.2), so these local tests might not mean much)

[[:file:Agriculture_characterisation_map,_Flanders.png]],

This one worked fine locally (It did require more than 100mb of memory, but still less than 400mb).

[[:file:Basemap_for_Afil.png]]

Seems to need a bigger temporary file (575 mb seemed to do it)

, [[:file:Drapeau_hillion.png]]

Seems to need a 738 mb temp file.


So it would appear increasing $wgMaxShellFileSize to say 1 GB would decrease the number of images we fail to resize by quite a bit. Not sure if that's something viable to do. At the very least, probably not until bug 52203 is fixed.

Doesn't block bug 9497 as that's closed.

(In reply to comment #1)

At the very least, probably not until bug 52203 is
fixed.

Well, if your bug 52203 comment 4 is correct, raising the limit to fix this bug would be a workaround for that bug too, because if uninterrupted the extension cleans up its garbage by itself.
The existing workaround now in place seems to make disk storage not a problem, but I'm sure that's not the only consideration.

(In reply to comment #3)

(In reply to comment #1)
> At the very least, probably not until bug 52203 is
> fixed.

Well, if your bug 52203 comment 4 is correct, raising the limit to fix this
bug
would be a workaround for that bug too, because if uninterrupted the
extension
cleans up its garbage by itself.
The existing workaround now in place seems to make disk storage not a
problem,
but I'm sure that's not the only consideration.

J had a tweak to the vips command line that should significantly reduce size of temp files (which should be deployed next week). That may stop many of the failures.

Proper solution to the temp file issue would maybe be to create temporary working directory for vips and delete everything in directory when done.

In regards to j's tweak - http://test.wikipedia.org/wiki/File:Huge.png is 160 mp and scaling fine now. True test will come when changes hit commons next week.

Ok, these are the biggest pngs (all pngs bigger than 170 MP) on commons:

MariaDB [commonswiki_p]> select img_name, img_width*img_height from image where img_media_type = 'BITMAP' and img_minor_mime = 'png' and img_major_mime = 'image' and img_width *img_height > 170000000 order by img_width*img_height desc;
+-----------------------------------------------------------+----------------------+

img_nameimg_width*img_height

+-----------------------------------------------------------+----------------------+

Iowa_County,_Wisconsin,_Courthouse_-_Icon_-_3000-dpi.png4161055734
Mandelbrot_fractal_large.png715776516
Mandelbrot_High_Resolution.png600000000
Iowa_County,_Wisconsin,_Courthouse_-_Icon_-1000-dpi.png462349287
Gonders_flats.png375955914
U-undStraßenbahnGleisplanBerlinundUmgebung2013.png293840000
Union_Stock_Yards,_Omaha,_1899.png260564824
Union_Stock_Yards,_Omaha,_1925.png259465592
Union_Stock_Yards,_Omaha,_1958.png258749920
Union_Stock_Yards,_Omaha,_1916.png258264072
Union_Stock_Yards,_Omaha,_1999.png257947776
Union_Stock_Yards,_Omaha,_1887.png257896800
Redstone_Rocket_Test_Stand_HAER_AL-129-A_sheet_1_of_7.png241231872
World4-isometric_north.png200950784
Rensselaerswyck_Original_Map_Large.png195843032
Drapeau_hillion.png192853712
Income_inequality_in_the_United_States,_1979-2007.png174420000

+-----------------------------------------------------------+----------------------+

enwiki has no pngs bigger than 170 MP

I tested the top 5, they all seem to render. I will note, that [[commons:file:Iowa_County,_Wisconsin,_Courthouse_-_Icon_-_3000-dpi.png]] doesn't seem to render the courthouse part of the image (compare with [[commons:file:Iowa_County,_Wisconsin,_Courthouse_-_Icon_-1000-dpi.png]]). I'm not sure if that's a problem with our rendering, or if the original image is just broken. I'm going to go with the original image is just broken unless someone comes up with evidence to the contrary.

In any case, marking this bug fixed

Gilles moved this task from Untriaged to Done on the Multimedia board.Dec 4 2014, 10:12 AM
Gilles raised the priority of this task from "Normal" to "Unbreak Now!".
hashar removed a subscriber: hashar.Dec 4 2014, 10:22 AM
Gilles lowered the priority of this task from "Unbreak Now!" to "Normal".Dec 4 2014, 11:20 AM

Add Comment