Thumbnails of large PNGs are not generated
Closed, ResolvedPublic

Description

Author: emddudley

Description:
The thumbnail images for large PNG images in galleries are not being generated
on Wikimedia Commons. An example is at the provided URL, where two images
([[:commons:Image:William-Adolphe Bouguereau (1825-1905) - The Broken Pitcher
(1891).png]] and [[:commons:Image:William-Adolphe Bouguereau (1825-1905) - The
Song of the Nightingale (1895).reduced.png]]) are not being rendered in the
1890s section gallery.

The two images are very large PNGs (over 13 and 17 MB). I have tried refreshing
the gallery page, purging the gallery page, and purging the image pages. Regular
thumbnails of the images display fine.

I'm not really sure if this is an issue with Mediawiki or with an image library...


Version: unspecified
Severity: major
URL: http://commons.wikimedia.org/wiki/File:World_TLD_Map.png
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=28135

bzimport set Reference to bz9497.
bzimport created this task.Via LegacyApr 4 2007, 7:28 PM
bzimport added a comment.Via ConduitApr 26 2007, 5:45 PM

emddudley wrote:

The images now display with the message "Error creating thumbnail: Invalid
thumbnail parameters", as on
http://commons.wikimedia.org/wiki/Image:William-Adolphe_Bouguereau_%281825-1905%29_-_The_Broken_Pitcher_%281891%29.png

brion added a comment.Via ConduitApr 26 2007, 7:56 PM

Both files are over the 12.5-megapixel default maximum for thumbnailing.

Large files (other than JPEG, which can be thumbnailed efficiently from larger
sizes) are currently prevented from being thumbnailed to avoid problems with too
much memory use on the servers.

Probably there should be a more useful error message here.

siebrand added a comment.Via ConduitOct 11 2007, 2:02 PM

I suggest we change [[MediaWiki:Thumbnail invalid params]] to "Invalid thumbnail parameters or non-JPG file with more than 12.5 million pixels" until a more memory efficient solution for rendering larger images has been found.

brion added a comment.Via ConduitOct 15 2007, 7:46 PM

Well, it'd be nice to distinguish between parameters that are "invalid" and "disallowed by policy". It's a matter of propagating the correct error condition back.

bzimport added a comment.Via ConduitApr 13 2008, 6:12 PM

Bryan.TongMinh wrote:

I'm working on a program that does not need to load the entire file in memory, see http://svn.wikimedia.org/viewvc/mediawiki/trunk/pngds/. It currently resizes PNGs, but outputs only in a raw RGB stream.

gpaumier added a comment.Via ConduitDec 31 2009, 12:25 AM
  • Bug 12374 has been marked as a duplicate of this bug. ***
AdamCuerden added a comment.Via ConduitApr 22 2010, 9:49 PM

Please, PLEASE do something about this. There's been a half-finished solution for ''TWO YEARS''.

bzimport added a comment.Via ConduitApr 23 2010, 2:12 AM

avarab wrote:

Maybe you should do something about it by contributing to the half-finished solution? Complaining on the bug tracker doesn't help.

Bawolff added a comment.Via ConduitApr 23 2010, 2:34 AM

I'm just curious, what remains to be done for pngds. I just tried it, it seemed to downsize large files fine, and outputs them as png.

(of course i didn't measure what resources it was using, nor do i know what resources would be acceptable. I did test it with [[File:Solar-system.png]] and it downsized it from 6,808 × 2,260 to 800x266 in 4.502 seconds (which doesn't matter as memory usage not execution time was the concerning resource from what i gather))

bzimport added a comment.Via ConduitApr 23 2010, 8:30 AM

Bryan.TongMinh wrote:

(In reply to comment #9)

I'm just curious, what remains to be done for pngds. I just tried it, it seemed
to downsize large files fine, and outputs them as png.

It works quite well, but there seems to be some random segmentation faults. This may very well be caused by the fact that it was the author's first program for the pc in C :) In addition I think it needs some work before it is actually reviewable.

MaxSem added a comment.Via ConduitApr 23 2010, 3:56 PM

I'll try to cleanup it. Bryan, do you still rememember how to reproduce those crashes?

bzimport added a comment.Via ConduitApr 23 2010, 4:40 PM

Bryan.TongMinh wrote:

(In reply to comment #11)

I'll try to cleanup it. Bryan, do you still rememember how to reproduce those
crashes?

No, they appeared randomly when a thumbnail was generated.

AdamCuerden added a comment.Via ConduitAug 25 2010, 7:06 PM

Any luck with this?

bzimport added a comment.Via ConduitNov 17 2010, 11:46 AM

test5555 wrote:

[[Commons:File:Nuevo_mapa_de_la_Republica_Argentina_(1914).png]] is a case mentioned at [[Commons:Commons:Administrators'_noticeboard#Too_much_is_tooooo_much]].

Permanent link: http://commons.wikimedia.org/w/index.php?title=Commons:Administrators%27_noticeboard&oldid=46170436#Too_much_is_tooooo_much

Dragons_flight added a comment.Via ConduitJan 24 2011, 6:45 AM

It has been pointed out to me that ImageMagick now includes command line options to limit memory usage (with the potential trade-off of using disk caching). See:

http://www.imagemagick.org/script/architecture.php#tera-pixel

for example options. Such options may provide a way to work with at least some large images (though caching to disk could also create other resource problems).

TheDJ added a comment.Via ConduitJul 12 2011, 9:25 PM
  • Bug 29818 has been marked as a duplicate of this bug. ***
Dcoetzee added a comment.Via ConduitAug 22 2011, 11:57 PM

I've created a new tool to help fix this bug, at:

https://github.com/dcoetzee/pngscale

My description:

pngscale is a specialized tool for scaling down of PNG files to create
thumbnails. It works scanline by scanline, so it's memory-efficient
even on extremely large PNG images, taking only about 11 MB of RAM in
experiments. It is also faster than ImageMagick convert, particularly
on large images, running about twice as fast on a 50 megapixel file.

It produces 24-bpp RGB thumbnails of palettized and 16-bit images and
8-bpp grayscale thumbnails of grayscale images. It has not been tested
with progressive or interlaced images, and cannot upscale images.
Error messages are currently English-only.

Compatibly licensed (MIT/X11). I have not done the work to integrate this tool into Mediawiki yet. If anyone has an interest and would like to do so that would be great - otherwise I can do it and post a patch here.

bzimport added a comment.Via ConduitAug 23 2011, 7:48 AM

Bryan.TongMinh wrote:

(In reply to comment #17)

I've created a new tool to help fix this bug, at:

Very nice. This is basically the same approach that I used when writing pngds a few years earlier. Wikimedia needs to determine what is the way forward. Aside from the two specialized PNG scalers in this bug, there is also VIPS (bug 28135) which would solve this problem.

MaxSem added a comment.Via ConduitAug 23 2011, 6:53 PM

Unassigning from myself, VIPS is the future of mankind.

Dcoetzee added a comment.Via ConduitAug 24 2011, 2:11 AM

Alright, guess it's up to me then. I will create a patch and attach it.

Dcoetzee added a comment.Via ConduitAug 27 2011, 5:49 AM

Created attachment 8978
A patch allowing custom scaling tools to be used for specific extensions.

This patch allows a line like this in LocalSettings.php to configure the use of a custom tool like pngscale for scaling PNGs on a Mediawiki server:

$wgCustomConvertCommandsByExtension['png'] = "/usr/bin/pngscale %s %d %w %h";

This supersedes other settings for files with the given extension. There may be multiple extension-specific convert commands. Default is array().

Attached: mediawiki_pngscale.patch

Dcoetzee added a comment.Via ConduitAug 27 2011, 5:53 AM

Also, I've updated pngscale now to handle upscaling of PNGs as well (with bilinear filtering), so it drops nicely into the patch I've just submitted. To test, I built pngscale from:

git checkout git://github.com/dcoetzee/pngscale.git

and installed in /usr/bin. I then configured my test Mediawiki server with:

$wgMaxShellFileSize = 100000000;
$wgCustomConvertCommandsByExtension['png'] = "/usr/bin/pngscale %s %d %w %h";

To use this on WMF servers they would have to be similarly configured. In tests it was indeed much faster than ImageMagick convert and used a lot less memory. Feedback on patch is welcome.

bzimport added a comment.Via ConduitSep 30 2011, 4:00 PM

sumanah wrote:

Added the "patch" and "need-review" keywords; Mark hopes to get someone to review the patch soon.

bzimport added a comment.Via ConduitSep 30 2011, 4:38 PM

Bryan.TongMinh wrote:

Like I said above, I think the proper way forward is to fix bug 28135.

Bawolff added a comment.Via ConduitSep 30 2011, 5:34 PM

Personally I don't see much point in the patch attached to this bug. You might as well just put a 2 line extension in LocalSettings.php using the BitmapHandlerTransform hook. (Unless comment 23 means get someone to review the thing on github as opposed to the patch attached to this bug)

Dcoetzee added a comment.Via ConduitOct 1 2011, 6:54 AM

I'm a newbie and don't know what the options are... I'm totally okay with throwing out the patch in favour of a simpler alternative implementation using existing mechanisms. Just wanted to get a proof of concept done that works and solves the problem.

bzimport added a comment.Via ConduitOct 20 2011, 12:04 PM

test5555 wrote:

Could we just have implement this fix implemented until the new feature is developed ?

bzimport added a comment.Via ConduitOct 20 2011, 12:32 PM

Bryan.TongMinh wrote:

This will hopefully be fixed real soonTM.
I hope that we can experimentally deploy VipsScaler in a few weeks, which will be able to render large PNGs. It depends a bit on review time on the WMF side.

bzimport added a comment.Via ConduitOct 22 2011, 12:34 PM

test5555 wrote:

It has been two months since Derrick posted a fix. Rather than wait longer, would someone review the patch and deploy it?

This way the bug would actually be fixed and further cookie licking prevented.

Dcoetzee added a comment.Via ConduitOct 23 2011, 12:20 AM

Note that it should be possible to deploy this without any patch at all, just by configuring the WMF servers correctly (at least according to Bawolff - I believe them). We might want to test configure it on the test wiki first. Where is the right place to look for someone to do that?

Bawolff added a comment.Via ConduitOct 23 2011, 6:24 PM

(In reply to comment #30)

Note that it should be possible to deploy this without any patch at all, just
by configuring the WMF servers correctly (at least according to Bawolff - I
believe them). We might want to test configure it on the test wiki first. Where
is the right place to look for someone to do that?

I don't know that much about procedures involving wmf server software deployment, but I imagine git://github.com/dcoetzee/pngscale.git would have to be reviewed first, and I imagine that it would take roughly the same amount of effort to review that than to review the VIPS thing. (These are just guesses though. I really don't know what I'm talking about in this area)

bzimport added a comment.Via ConduitNov 7 2011, 3:34 AM

sumanah wrote:

For reference, the wikitech-l (developers' mailing list) thread about the upcoming VipsScaler deployment: http://lists.wikimedia.org/pipermail/wikitech-l/2011-November/thread.html#56064

TheDJ added a comment.Via ConduitAug 23 2012, 4:41 PM
  • Bug 18826 has been marked as a duplicate of this bug. ***
Nemo_bis added a comment.Via ConduitOct 27 2012, 7:39 PM

(In reply to comment #32)

For reference, the wikitech-l (developers' mailing list) thread about the
upcoming VipsScaler deployment:
http://lists.wikimedia.org/pipermail/wikitech-l/2011-November/thread.html#56064

I think it can be considered dead now, implementing pngscale as an extension (like wikidiff2) or whatever seems definitely the way to go.
Note that bug 41125 reduced the affected PNGs on Commons by an order of magnitude (from 12569 to 1528 as of the 24th).

AdamCuerden added a comment.Via ConduitNov 29 2012, 10:37 AM

This has been *FIVE YEARS* waiting for a fix. Seriously, this is just grossly incompetent now.

Bawolff added a comment.Via ConduitNov 29 2012, 9:19 PM

(In reply to comment #35)

This has been *FIVE YEARS* waiting for a fix. Seriously, this is just grossly
incompetent now.

{{sofixit}}

[Yes there's a patch on the bug. It still needs to be reviewed by somebody other than the author, tested, etc. These are all things you could have been doing in the last 5 years].

AdamCuerden added a comment.Via ConduitNov 30 2012, 11:09 AM

Yes, because everyone has computer algorithm expertise, Bawolff. Why don't you restore images, like I do? I mean, people can magically get new talents, according to you.

This is a basic functionality of Wikipedia. Wikipedia should, if necessary, be paying people to maintain their basic functionality if the volunteers are insufficient.

Bawolff added a comment.Via ConduitNov 30 2012, 8:52 PM

(In reply to comment #37)

Yes, because everyone has computer algorithm expertise, Bawolff. Why don't you
restore images, like I do? I mean, people can magically get new talents,
according to you.

I apologize if my tone was inappropriate in my previous comment. My point was, that for example if there was an image that I really wanted restored - I would either learn how to restore images, politely convince someone else to restore it for me, or live with the image not being restored. If I really felt strongly about it, I might even pay someone to restore the image for me. I would not yell at the people doing restorations that they are incompetent simply because they have different priorities for what they feel are important or otherwise do things other then what I want them to do with their free time.

Qgil added a comment.Via ConduitDec 17 2012, 5:03 PM
  • Bug 42531 has been marked as a duplicate of this bug. ***
Qgil added a comment.Via ConduitDec 17 2012, 5:17 PM

fwiw I have listed this bug at https://www.mediawiki.org/wiki/Annoying_large_bugs (without knowing whether it is actually large or not).

The duplicated bug report contains this comment from Adam that is worth having here as well:

[[:File:Gustave_Doré_-_Dante_Alighieri_-_Inferno_-_Plate_9_(Canto_III_-_Charon).png]]

[[:File:W.E.F._Britten_-_The_Early_Poems_of_Alfred,_Lord_Tennyson_-_Oenone.png]]

[[:File:Gavin_Hamilton_-_Coriolanus_Act_V,_Scene_III.png]]

[[:File:Louis_Huard_-_The_Punishment_of_Loki.png]]

[[:File:Ulysses_S._Grant_from_West_Point_to_Appomattox.png]]

That's a small smattering of the Featured Pictures I had to upload twice, once
as a lossy JPEG, once as a lossless PNG.

It's actually considered a requirement by some at English Featured Picture
Candidates that the PNG of a restoration be uploaded alongside the JPEG that
Wikipedia actually allows users to use. See, for example:

[[:en:Wikipedia:Featured_picture_candidates/The_Early_Poems_of_Alfred,_Lord_Tennyson]]
"PNG versions, could you upload PNG versions for a lossless version if someone
else wants to use or edit them? Are the original versions uploaded too?"


I agree this bug is painful but if nobody is looking at it is just because of a lack of resources, not because of some kind of mania or malice. I wonder whether we have developers in the community with the skills to loo at this problem & patch, and whether they are aware of this report.

bzimport added a comment.Via ConduitDec 22 2012, 5:59 PM

sumanah wrote:

Nemo, can you explain how you figured out how many files are affected? That will affect how highly we should prioritize this bug.

(In reply to comment #3)

I suggest we change [[MediaWiki:Thumbnail invalid params]] to "Invalid
thumbnail parameters or non-JPG file with more than 12.5 million pixels"
until
a more memory efficient solution for rendering larger images has been found.

On en.wp, the message is "Invalid thumbnail parameters or image file with more than 12.5 million pixels", and on Commons it is now "Invalid thumbnail parameters or PNG file with more than 25 million pixels". So we probably don't need to change the message in MessagesEn.php directly. :)

I've updated https://www.mediawiki.org/wiki/VipsScaler and bug 28135 (the review-and-deploy-VipsScaler issue); honestly, it sounds to me like people should consider moving forward with a non-VipsScaler solution in the short term to fix this particular bug.

Does the patch in attachment 8978 in comment #21 still work?

(In reply to comment #31)

(In reply to comment #30)
> Note that it should be possible to deploy this without any patch at all, just
> by configuring the WMF servers correctly (at least according to Bawolff [comment #25] - I
> believe them). We might want to test configure it on the test wiki first. Where
> is the right place to look for someone to do that?

I don't know that much about procedures involving wmf server software
deployment, but I imagine git://github.com/dcoetzee/pngscale.git would have
to
be reviewed first, and I imagine that it would take roughly the same amount
of
effort to review that than to review the VIPS thing. (These are just guesses
though. I really don't know what I'm talking about in this area)

We're trying to update the documentation around WMF server software deployment changes -- see https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment and https://www.mediawiki.org/wiki/Developers/Maintainers#Operations.2Fsystems_administration . If you just need to figure out who to ask or how to move forward on a request, the #wikimedia-operations and #wikimedia-tech channels have a rotation of which Operations person is "on RT duty" -- http://wikitech.wikimedia.org/view/Interrupts_Rotation has more information and a listing.

Dcoetzee added a comment.Via ConduitDec 22 2012, 8:17 PM

Last patch review they determined that the patch was unnecessary because my pngscale tool could actually be deployed with existing mechanisms with little trouble (all the patch did was provided a mechanism to apply a particular rescaling tool to PNGs while using a different one for other files - if nothing else this would be simple to do on existing deployments by using a script as the rescaling tool that invokes other tools as needed).

I do imagine it would be necessary for someone else to review my tool to make sure it doesn't have serious security problems that I'm unaware of, and to make sure my test suite is thorough enough. But other than that deploying this on existing Mediawiki including WMF servers should be straightforward.

Bawolff added a comment.Via ConduitDec 23 2012, 1:43 AM

On en.wp, the message is "Invalid thumbnail parameters or image file with
more
than 12.5 million pixels", and on Commons it is now "Invalid thumbnail
parameters or PNG file with more than 25 million pixels". So we probably
don't
need to change the message in MessagesEn.php directly. :)

Actually, the thumbnail parameter error message is used to signal several error types. The message used to report this error should be re-worked to be specific to the actual error (or at least common cases should have their own message). However that's an entirely different issue. [On a similar note, the message given to the API should also be a unique error identifier, not the message text]

McZusatz added a comment.Via ConduitMay 2 2013, 12:13 PM

$wgMaxImageArea and $wgMaxAnimatedGifArea is now at 50 MP. See bug 47363 .

Kelson added a comment.Via ConduitMay 2 2013, 1:17 PM

@Marco

Really nice. Thank you a lot for that.

Now, a lot of our pictures of the Zurich central library have thumbnails.

But, we still have a few one over the limit:
https://commons.wikimedia.org/wiki/File:Zentralbibliothek_Z%C3%BCrich_-_Vue_perspective_de_la_ville_du_lac_et_environs_de_Zuric_prise_%C3%A0_lAuberge_de_lEp%C3%A9e_du_Cot%C3%A9_de_lOrient_-_000008012.tif

This forces us to still upload both a JPG version and a TIFF version.

Bawolff added a comment.Via ConduitJul 25 2013, 9:11 PM

Update. Large png images can now be scaled. (There is still a limit. I'm not sure precisely what it is, but roughly speaking around 140 megapixels, stuff stops working). - bug 52050.

Note tiff files still have the old limit.

This bug should probably either be closed, or turned into a tracking bug for all "scaling large images" issues.

Kelson added a comment.Via ConduitJul 26 2013, 7:02 AM

Should I open a bug specific for TIFF files? Our GLAM partner pictures are still not all thumbnailed correctly?
https://commons.wikimedia.org/wiki/Category:Media_contributed_by_Zentralbibliothek_Z%C3%BCrich_%28original_picture%29

Bawolff added a comment.Via ConduitJul 26 2013, 5:37 PM

(In reply to comment #47)

Should I open a bug specific for TIFF files? Our GLAM partner pictures are
still not all thumbnailed correctly?
https://commons.wikimedia.org/wiki/Category:
Media_contributed_by_Zentralbibliothek_Z%C3%BCrich_%28original_picture%29

There's already one at Bug 52045.

Aklapper added a comment.Via ConduitJul 30 2013, 11:27 AM

Closing as per comment 46.

Gilles added a project: Multimedia.Via WebDec 4 2014, 10:15 AM
Gilles raised the priority of this task from "High" to "Unbreak Now!".Via WebDec 4 2014, 10:21 AM
Gilles moved this task to Closed on the Multimedia workboard.
Gilles lowered the priority of this task from "Unbreak Now!" to "High".Via ConduitDec 4 2014, 11:21 AM
Kozuch awarded a token.Via WebDec 17 2014, 8:02 PM

Add Comment