Page MenuHomePhabricator

Greyscale pngs without gAMA chunk rendered with incorrect contrast [or setting the gamma in GIMP exports to PNG]
Open, LowPublic

Tokens
"The World Burns" token, awarded by Ahecht."The World Burns" token, awarded by Poyekhali."The World Burns" token, awarded by Revent."The World Burns" token, awarded by zhuyifei1999."The World Burns" token, awarded by Natuur12."The World Burns" token, awarded by Josve05a."The World Burns" token, awarded by Steinsplitter.
Assigned To
Authored By
Bawolff, Jul 22 2015

Description

It appears that image magick treats png greyscale files as being linear greyscale, instead of the normal srgb-ish gamma of 2.2

Probably a bug in image magick. I Think its fixed in 6.9.0-1 but haven't tested [Edit: I've now tested and can confirm upgrade would fix] ( See http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=26576 ). So probably we just need to upgrade.

https://commons.wikimedia.org/w/index.php?title=Commons:Village_pump&oldid=166488123#Image_rendering_issues
https://commons.wikimedia.org/w/index.php?title=Commons:Graphics_village_pump&oldid=166489831#PNG_darkness_at_preview_sizes

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Bawolff updated the task description. (Show Details)Jul 22 2015, 1:04 PM

Change 226665 had a related patch set uploaded (by Brian Wolff):
Workaround im bug where greyscale have assumed gamma of 1

https://gerrit.wikimedia.org/r/226665

I just tested on most recent image magick. Can confirm that upgrading image magick would fix the issue.

As T84842 is solved, can image magick be upgraded now to a recent enough version?

zhuyifei1999 moved this task from Incoming to Backlog on the Commons board.Aug 9 2015, 8:19 AM

As T84842 is solved, can image magick be upgraded now to a recent enough version?

Normally it's the version and package that the distribution (Ubuntu or Debian) provides. Except for good reasons to change code. Not sure if this is important enough that it justifies maintain our own package / diff.

Jdforrester-WMF triaged this task as Low priority.Sep 4 2015, 6:55 PM
Jdforrester-WMF moved this task from Untriaged to Backlog on the Multimedia board.Sep 4 2015, 6:59 PM
Yann raised the priority of this task from Low to Normal.Oct 3 2015, 12:20 PM
Aklapper lowered the priority of this task from Normal to Low.Oct 4 2015, 9:27 AM

@Yann: Do you plan to work on fixing this or why did you raise the priority of this task?

Yann added a comment.Oct 4 2015, 12:36 PM

@Aklapper I hope that being a developer is not a new standard to request a bug to be fixed in a reasonable time scale?

Krenair added a subscriber: Krenair.Oct 4 2015, 1:06 PM

How is raiasing task priority "requesting a bug to be fixed in a reasonable time scale"?

Let's not play on words, it's clear that priority reflects how important a task is. Do we know how many files are affected?

Let's not play on words, it's clear that priority reflects how important a task is. Do we know how many files are affected?

Is there even a way to query that?

It depends how you define "query". :)

$ curl -s 'https://upload.wikimedia.org/wikipedia/commons/archive/5/55/20151003120942%21Abraham_Lincoln_by_Alexander_Hesler.png' | exiftool -fast2 -
ExifTool Version Number         : 10.00
File Type                       : PNG
File Type Extension             : png
MIME Type                       : image/png
Image Width                     : 1208
Image Height                    : 1762
Bit Depth                       : 8
Color Type                      : Grayscale
Compression                     : Deflate/Inflate
Filter                          : Adaptive
Interlace                       : Noninterlaced
Background Color                : 255
Pixels Per Unit X               : 2835
Pixels Per Unit Y               : 2835
Pixel Units                     : meters
Modify Date                     : 2015:10:03 12:03:44
Image Size                      : 1208x1762
Megapixels                      : 2.1

Are all grayscale PNGs affected?

MariaDB [commonswiki_p]> select img_metadata from image where img_name = 'Abraham_Lincoln_by_Alexander_Hesler.png';
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| img_metadata                                                                                                                                                             |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| a:6:{s:10:"frameCount";i:0;s:9:"loopCount";i:1;s:8:"duration";d:0;s:8:"bitDepth";i:8;s:9:"colorType";s:10:"truecolour";s:8:"metadata";a:1:{s:15:"_MW_PNG_VERSION";i:1;}} |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Maybe:

MariaDB [commonswiki_p]> select count(img_sha1) from image where img_metadata LIKE '%s:8:"bitDepth";i:8;s:9:"colorType";s:10%';
+-----------------+
| count(img_sha1) |
+-----------------+
|          316217 |
+-----------------+
1 row in set (4 min 54.69 sec)

@Aklapper I hope that being a developer is not a new standard to request a bug to be fixed in a reasonable time scale?

@Yann: If you'd like to request fixing in some time scale, please add a comment suggesting the change and convincing reasons. Thank you!

Steinsplitter added a subscriber: Josve05a.
Yann added a comment.Oct 18 2015, 9:54 PM

@Aklapper This is a serious issue. It affects a lot of files, and everybody including readers and contributors. This seems sufficient to me for this bug to be fixed quite soon.

Josve05a moved this task from Backlog to Patch merged upstream on the Upstream board.
Josve05a moved this task from Backlog to Doing on the good first bug board.
Josve05a moved this task from Doing to Backlog on the good first bug board.Oct 18 2015, 10:17 PM
Tgr added a subscriber: Tgr.Oct 19 2015, 6:04 AM

So what exactly is needed to fix this? Either upgrade IM to 6.9.1+ or merge Bawolff's patch?

So what exactly is needed to fix this? Either upgrade IM to 6.9.1+ or merge Bawolff's patch?

Is there any harm in upgrading? 6.9.0 is a fairly old version dating back to March.

Wikimedia's servers run Ubuntu and Debian; Ubuntu ships/provides IM version 6.7.7. Hence there is no harm but there are maintenance costs.

Hmm. That version dates back to March 2014. Is there a way to ask the ubuntu packagers to upgrade their package?

Hmm. That version dates back to March 2014. Is there a way to ask the ubuntu packagers to upgrade their package?

That's not how Ubuntu works. If we cared about always having recent software, we'd run a different distro. :)

Natuur12 added a subscriber: Natuur12.
Revent added a subscriber: Revent.Nov 3 2015, 10:11 PM
Restricted Application added a project: Multimedia. · View Herald TranscriptJan 26 2016, 4:18 PM

Change 226665 abandoned by Brian Wolff:
Workaround im bug where greyscale have assumed gamma of 1

Reason:
Abandoning this for now. I'm not going to work on this in the near-future. Someone else will have to take it up.

https://gerrit.wikimedia.org/r/226665

While we're waiting for someone else to take this up, a workaround on a per-image basis is to download the image, convert it to grayscale using a more recent version of Imagemagick on your local computer (using "convert {input_image} -type GrayScale -depth 8 gray.png", and re-uploading it. It would be nice, however, if this were fixed on the server somehow.

@Aklapper @brion surely it is reasonable that we get this fixed. Some of us just use plain old image editors and trying to install software and running command line fixes like this just seems unreasonable.

upload version https://upload.wikimedia.org/wikipedia/commons/1/12/Lady_Peel.png
thumbnail version https://commons.wikimedia.org/wiki/File:Lady_Peel.png (pale and washed out)

Eight months later it seems reasonable that it gets fixed, quality is important when reproducing images.

Would you volunteer to compile and maintain a custom package to deploy on servers?

This doesnt need a custom compiled version of image magick. If someone wants to address the code review comments on the abandoned patch, that will fix it. I just dont have plans to work on that patch in near future, so I abandoned it.

I volunteer to do lots of things and have for years. So please don't take
that path.

I know my capabilities, and this isn't one of them. So that is neither a
reasonable nor practical solution. Not sure that it is even valuable as
rhetoric, and it wasn't helpful.

I also know that grey scale pngs are not unexpected especially when
reproducing 19th century printed works. I am also told that these images
shouldn't be reproduced as lossy jpgs. So do nothing may be practical, but
not clearly reasonable.

While for "new" pngs, there is a solution for the Linux technorati which is
reasonable, it is not practical for the vast bulk of users. It isn't
practical for all the existing images.

If someone was to set up a bot to hunt and convert existing and new
"problematic" images, that is reasonable and maybe practical for someone.
It may be a reasonable first aid if there is no system fix, but it is
reactive and not a fix of the root cause.

After eight months to still have "do nothing but stare at it" as the
solution is an issue. So maybe we have to talk about a less than perfect
solution of what is possible, rather than do nothing.

End users have no control, whereas you developers have complete control of
the resouces and the solution. So please can we look for a solution and if
that means a custom build for a while let the tech heads have that
discussion. Or that the organisation uses its partnership relations to have
the issue fixed at source. (Clearly some politics there that I don't
understand.)

This comment was removed by Storkk.
This comment was removed by Billinghurst.

@Storkk your last assertion isn't quite true, FYI

I removed my comment in order to attempt to avoid derailing this bug report, which I have failed to do. Apologies to all for that.

I am happy that @Bawolff corrected me in T106516#2073472.
So anybody strongly interested in seeing this fixed will have to convince a developer to improve https://gerrit.wikimedia.org/r/226665 .

brion added a comment.Mar 1 2016, 4:49 PM

@Bawolff I'm still a bit swamped with recent non-code stuff but I'll make sure it's in my review queue, will try to take a look over it soon. If anybody else gets to it sooner, awesome. :)

Change 226665 restored by Brion VIBBER:
Workaround im bug where greyscale have assumed gamma of 1

Reason:
(reopening to switch ownership and rebase)

https://gerrit.wikimedia.org/r/226665

brion claimed this task.Mar 2 2016, 8:10 PM

(self-assigning)

Poyekhali added a subscriber: Poyekhali.

I guess this is the only task that has a lot of "The World is Burning" token...

I'm still a bit swamped with recent non-code stuff but I'll make sure it's in my review queue, will try to take a look over it soon.

@brion: Any luck with that? :) https://gerrit.wikimedia.org/r/#/c/226665/

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptMay 31 2016, 8:13 AM

Are we any closer to getting a permanent resolution to this?

I have just uploaded a batch of images like

https://commons.wikimedia.org/wiki/File:An_anxious_time_near_goal._%22Shoot!%22.png
(pale and washed out) compared to the upload
https://upload.wikimedia.org/wikipedia/commons/c/cc/An_anxious_time_near_goal._%22Shoot%21%22.png

and for me the problem still exists. I am using GIMP, so even if someone can tell me what I should do differently with GIMP. Thanks.

I have interrogated GIMP, and I will now export from GIMP forcing the gamma setting (which GIMP indicates shouldn't need to be set)

I have interrogated GIMP, and I will now export from GIMP forcing the gamma setting

Interesting. Let's document this workaround somewhere. Maybe https://commons.wikimedia.org/wiki/Commons:File_types#PNG ? Or at least the task description.

Billinghurst renamed this task from Greyscale pngs without gAMA chunk rendered with incorrect contrast to Greyscale pngs without gAMA chunk rendered with incorrect contrast [or setting the gamma in GIMP exports to PNG].Dec 4 2016, 10:54 AM

Is there still no way of a permanent fix? I have been using GIMP with forced gamma settings for a while and even reuploaded lots of my older images for this reason. But even that didn't work with some of my recent uploads, e.g. https://commons.wikimedia.org/wiki/File:Andr%C3%A9_Campra_-_Idom%C3%A9n%C3%A9e_-_title_page_of_the_score_-_Paris_1712.png - the jpg version of that image looks much better in the overview: https://commons.wikimedia.org/wiki/Category:Idom%C3%A9n%C3%A9e .

Rsteen added a subscriber: Rsteen.Nov 15 2017, 12:09 PM

This problem is not just GIMP-related. I regularly make PNG files with the FastStone Image Viewer - typically from converted jp2-files - and it irritates me considerably that they get such a pale appearance in the thumbs. What can be done?

This problem is not just GIMP-related. I regularly make PNG files with the FastStone Image Viewer - typically from converted jp2-files - and it irritates me considerably that they get such a pale appearance in the thumbs. What can be done?

It has never said to be a gimp issue, it is a known Wikimedia issue. That said there is a means for GIMP to export PNG files that do not suffer the identified issue. I would hazard a guess that your software has a solution too. See if there is a means to force the gamma setting.

Framawiki moved this task from Backlog to Doing on the good first bug board.Dec 2 2017, 1:33 PM

AFAICT production is using the expected, "normal" version of ImageMagick for our variant on Debian Stretch: 6.9.7-4.

Did the patch that fixes this in 6.9.0-1 get released into 6.9.7-4? If so, that means that any new thumbnails are being created OK but the old cached thumbs are wrong. So to fix this we'd need to delete the wrong ones and have the image scalers re-generate them. To avoid a stampede, SRE would need to do something special to identify which to delete and iterate slowly. Not sure if such code exists? (Fastest option)

If not, someone could ask the Debian ImageMagick maintainers to back-port it into their custom release for Debian Stretch, at which point production would get eventually get upgraded (and the file regeneration above would then have to happen). (Medium option)

If not, we'd need to move Wikimedia production's version of ImageMagick away from the widely-tested and security-supported upstream version with a custom backport, which is generally a Bad Idea™. I don't know if SRE have the bandwidth to do that? (Slow, hardest option)

If not that, we'd need to wait for Wikimedia production to move to Debian Buster. I don't imagine that's going to happen until early 2020 at the earliest; there's not even a planning ticket for that yet — it's not expected for Debian to declare it a stable release until mid-2019. (Slowest option)

Change 226665 abandoned by Brian Wolff:
Workaround im bug where greyscale have assumed gamma of 1

Reason:
I'm not working on this anymore

https://gerrit.wikimedia.org/r/226665