Page MenuHomePhabricator

Transparent background renders as white in PNG thumbnails
Closed, ResolvedPublic

Tokens
"Doubloon" token, awarded by RP88."Cup of Joe" token, awarded by MusikAnimal."The World Burns" token, awarded by Thibaut120094."Heartbreak" token, awarded by Kaartic."Burninate" token, awarded by BethNaught."The World Burns" token, awarded by Jack_who_built_the_house."The World Burns" token, awarded by Ahecht."Burninate" token, awarded by SerDIDG.
Assigned To
Authored By
XYZtSpace, Jun 28 2018

Description

PNG images with transparent backgrounds, when resized by the Mediawiki software, have the transparency replaced by opaque white.

For example, Fac-tory-o's N1 rocket render (Used in the N1 Wikipedia article's infobox) is shown with a white background in the infobox, which uses 100px-N1_Rocket.png, but it appears transparent in the Media Viewer, which uses the un-resized N1_Rocket.png, or when viewing the original full-size file on Commons..

Wikimedia Commons file link. The thumbnails in the Commons page also have white backgrounds.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Ping @Gilles, maybe he knows what changed.

Clpo13 added a subscriber: Clpo13.Jul 16 2018, 10:20 PM

I checked software updates deployed on the Thumbor hosts, but none of the multimedia lib updates add it. Timing-wise the most likely source of error is the python-thumbor-wikimedia update which was deployed on the 26th of June. I'll investigate this further tomorrow.

I looked into the changes between python-thumbor-wikimedia 1.16 and 2.0 and the most likely cause for this bug are the changes made for T27611:
https://github.com/wikimedia/operations-debs-python-thumbor-wikimedia/commit/23b589a5b56257e37c82840a3e091cceae72a959

Another case where this happens is this image:
https://de.wikipedia.org/wiki/Datei:Logo_PeerTube.png

here the transparent background gets black:

Glrx added a subscriber: Glrx.Jul 26 2018, 9:20 PM
Lofhi added a subscriber: Lofhi.Jul 27 2018, 7:52 PM
Kaartic added a subscriber: Kaartic.

This image successfully retains the transparent background layer when previewed on its File: page, but when used in the article it was designated for, the transparent layer still does not appear, and it is still a solid white.

MusikAnimal added a subscriber: MusikAnimal.

(unsure if this is related)

This image is in .png and has a transparent background when the original is opened directly, but a white background in the media viewer and infobox. It also has a white background in all versions other than the largest, original file.
https://upload.wikimedia.org/wikipedia/commons/6/60/SpaceX_CRS-15_Patch.png

RP88 added a subscriber: RP88.Aug 22 2018, 11:17 PM
RP88 awarded a token.Aug 22 2018, 11:20 PM

Sorry, can anybody explain why images from https://commons.wikimedia.org/wiki/Category:Transparent_squares are not affected? Look e. g. at https://upload.wikimedia.org/wikipedia/commons/thumb/0/02/Transparent_square.svg/512px-Transparent_square.svg.png – the SVG⇒PNG conversion occurred today, the PNG wasn’t cached from þe olde goode times, and it is fully transparent. Some essential conditions appear to be missing from description of the bug.

Oops, from https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29/Archive_167#Transparent_png_files and other discussions this is irrelevant to SVG at all, but something with Web typesetting by MediaWiki. Missed the point.

Jeff_G added a subscriber: Jeff_G.Aug 26 2018, 3:41 AM
Gilles triaged this task as Normal priority.
Gilles added a project: Thumbor.
PhilipTerryGraham raised the priority of this task from Normal to High.Aug 27 2018, 4:02 PM

This is a fundamental flaw in the image system. It still hasn't been fixed, and in the interest of having properly-rendered PNGs on Wikimedia after almost two months without such, it really needs to be fixed soon... :/

It appears that "High" has been redefined to "currently working on." So then I must ask, how is one supposed to indicate that a project should be higher priority if you've redefined "High" to mean "currently working on"? I don't see a way to vote on a bug.

It is quite surprising to me that this has taken so long. This is a regression, and the update that caused it has been identified. Yet there has been no talk of simply pulling out the change until it can be reapplied without breaking anything. I'd get it if that's not possible because of other changes or security issues, but nothing like that has been proposed.

Is there any reason you don't just roll back to Python 1.16 until you can figure out how to get this working on Python 2.0?

Once this is resolved, is there any way to programmatically regenerate all PNG thumbnails created since the change to python-thumbor-wikimedia 2.0?

Oops, I don't know how those projects got removed...

@Aklapper I wouldn't know how to fix this, I'm just a Wikipedia editor that was pointed to this task by fellow editors about a month ago. I'm simply trying to express the most amount of frustration with the least amount of repercussions towards the fact that even after two months, 30 subscribers to the task, and potentially hundreds broken PNG files strewn about Wikimedia, work towards fixing this relatively major flaw seems non-existent. I'm sorry if I crossed a line or two, but I feel like two months of sitting on something this major is too long. If there's a better way to express frustration and prompt more immediate action in situations like this, be sure to let me know.

TheDJ added a comment.Aug 28 2018, 8:31 PM

I'm sorry if I crossed a line or two, but I feel like two months of sitting on something this major is too long. If there's a better way to express frustration and prompt more immediate action in situations like this, be sure to let me know.

Please appreciate that there are a lot of bugs, few people to work on these, fewer people with knowledge of this particular area and even fewer people during a vacation period.
This is a non-destructive problem, so it's priority is relatively low compared to the many other day-to-day problems.

Gilles added a comment.EditedAug 29 2018, 7:52 AM

I've found the source, it's a change in the resizing/cropping logic (old logic was incorrect in specific cases) that triggers what is likely to be an IM bug on Debian Jessie:

convert -define tiff:exif-properties=no -resize 400x300 -quality 95 -sampling-factor 4:2:0 ~/thumbor-plugins/tests/integration/originals/PNG_transparency_demonstration_1.png png:- > old.png

correct transparency

convert -define tiff:exif-properties=no -resize 400x300^ -gravity center -extent 400x300 -quality 95 -sampling-factor 4:2:0 ~/thumbor-plugins/tests/integration/originals/PNG_transparency_demonstration_1.png png:- > new.png

white background

The newer IM version I have locally isn't affected (same command as above running on OS X/IM 7.0.8-2):

correct transparency

This was actually caught by existing tests during the big refactor, but I overrode the value when the test failed on Jessie (it passed locally), assuming it was something to do with the compression of the alpha layer on the IM version we have running in production, when in fact the alpha layer was just not transparent at all.

Either I find a workaround, or PNGs will have to live with the slightly incorrect resizing/cropping in edge cases we used to have until we upgrade Debian/IM on the Thumbor hosts.

Correct transparency is more critical than off-by-one-pixel sizes (with the simpler IM old IM resizing command that retains transparency on Jessie, images can sometimes be 1px wider or narrower than requested).

Change 456145 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.01

https://gerrit.wikimedia.org/r/456145

Change 456145 merged by Filippo Giunchedi:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.1

https://gerrit.wikimedia.org/r/456145

Since potentially tens of thousands differrnt sized thumbnails were affected by this bug, will the server eventually regenerate all of these, or will they need to be purged manually?

@Ahecht it seems to be fixing itself... rather slowly. Most of the files I've uploaded when the transprency layer bug was around now have their transparency layers in thumbnails, but some still have the opaque background. For example, the lead images in the Microsoft Office 2019 and Microsoft Word articles on the English Wikipedia now appear as intended, but the image on the Microsoft Excel article still has an opaque background; my browser cache has been cleared and both the article and file page have been purged. Though, I suspect this will be fixed naturally as the server catches up with all these files.

They won't necessarily all fix themselves, no, this is why I mentioned that they need purging to ensure that they're fixed. I can purge all PNGs, but that's going to be quite time-consuming.

@Gilles can the everyday Wikipedia user such as myself help in the process of purging these PNGs? The usual "?action=purge" URL technique doesn't seem to work.

That's action=purge yes. After you've done that, you need to make sure that you clear your browser cache or to shift-refresh the page where the thumbnail is, otherwise you'll keep seeing the old incorrect thumbnail coming from your browser cache.

@Gilles I should also clarify, just as I had already said earlier, the opaque backgrounds still appear even after my browser's cache was cleared.

@Gilles I should also clarify, just as I had already said earlier, the opaque backgrounds still appear even after my browser's cache was cleared.

Please always provide specific examples so others can reproduce (or not). Which browser? Also see https://en.wikipedia.org/wiki/Wikipedia:Bypass_your_cache

@Aklapper Firefox Nightly 63.0, Windows 10 version 1803, Ctrl+F5 doesn't work either; I've been trying on the Microsoft Excel English Wikipedia article to get the lead image to show up with a transparent layer with all three aforementioned methods.

Ah, you need to purge the file itself, not the article it appears in. I.e. https://en.wikipedia.org/wiki/File:Microsoft_Excel.png?action=purge

I'll let you try it yourself, I didn't purge it.

@Gilles This works! Thanks for the tip; I'll go ahead and purge as many PNGs as I can!

@Gilles This works! Thanks for the tip; I'll go ahead and purge as many PNGs as I can!

You can try T170039#3473755.

Doesn't seem to be completely fixed: see

Both are PNG8 (aka Indexed PNG) if that's relevant.

Gilles reopened this task as Open.EditedSep 17 2018, 6:00 PM

It's probably relevant that they're using indexed colors, indeed.

Change 461793 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.2

https://gerrit.wikimedia.org/r/461793

Change 461793 merged by Filippo Giunchedi:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.2

https://gerrit.wikimedia.org/r/461793

Stashbot added a subscriber: Stashbot.

Mentioned in SAL (#wikimedia-operations) [2018-09-24T09:30:06Z] <godog> upgrade / roll restart thumbor in eqiad / codfw - T20871 T198370

Gilles closed this task as Resolved.EditedSep 24 2018, 9:36 AM

Palette PNGs are now fixed as well, I purged those 2 examples you gave @Trlkly and verified the fix. Thanks again for your report.

Like before, purge any affected PNG for it to get fixed thumbnails.

Anomie added a subscriber: Anomie.Feb 3 2019, 6:31 PM

How about "Greyscale with Alpha" images? https://commons.wikimedia.org/wiki/File:Quillette.png seems to still be getting the white background as described in this task.

Anomie added a comment.Feb 3 2019, 7:39 PM

Also it seems to do the white background with RGB and greyscale images using the tRNS chunk rather than a full alpha channel:

Thanks for the test files, the fix looks straightforward at a glance.

Change 487785 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/software/thumbor-plugins@master] Fix PNG transparency for more cases

https://gerrit.wikimedia.org/r/487785

Change 487785 merged by Gilles:
[operations/software/thumbor-plugins@master] Fix PNG transparency for more cases

https://gerrit.wikimedia.org/r/487785

Change 488037 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/software/thumbor-plugins@master] Version bump

https://gerrit.wikimedia.org/r/488037

Change 488037 merged by Gilles:
[operations/software/thumbor-plugins@master] Version bump

https://gerrit.wikimedia.org/r/488037

Change 488060 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.3

https://gerrit.wikimedia.org/r/488060

Change 488060 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.3

https://gerrit.wikimedia.org/r/488060

Change 488060 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.3

https://gerrit.wikimedia.org/r/488060

Change 488060 merged by Gilles:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.3

https://gerrit.wikimedia.org/r/488060