Page MenuHomePhabricator

Issues with displaying thumbnails for CMYK JPG images due to buggy version of ImageMagick (black horizontal stripes, black color missing)
Closed, ResolvedPublic

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added subscribers: Poyekhali, Steinsplitter, Matanya, Aklapper. · View Herald TranscriptJul 31 2016, 8:56 PM
Fae awarded a token.Sep 30 2016, 11:04 AM
Fae added a subscriber: Fae.Sep 30 2016, 11:07 AM

More examples from different sources:

For batch uploads this presents a bit of a nightmare as the badly thumbnailed files are hard to detect without eating up a lot of bandwith.

matmarex claimed this task.Oct 5 2016, 6:09 PM
matmarex added a subscriber: matmarex.

I'll try to look into this.

Gilles added a subscriber: Gilles.Oct 7 2016, 1:18 PM

Our current version of Thumbor+plugins in production handles those fine:

Since the plan is to have Thumbor handle all thumbnail production traffic at some point this quarter, this task can probably wait for that to happen. If you want to fix this in Mediawiki's thumbnailing code, the first thing I'd check is ImageMagick versions and trying to run the same underlying IM command locally. This feels like an IM bug at first glance and it might just be a case of the Thumbor boxes having a more recent IM than the image scalers.

matmarex renamed this task from Issue with displaying thumbnail for a progressive CMYK-JPG image to Issues with displaying thumbnails for CMYK JPG images.Oct 11 2016, 8:43 PM
matmarex added subscribers: FDMS, Platonides.

OK, so I'm going to start by finding out which versions are affected. For a start, I reproduced locally, so this is not specific to the production configuration.

MediaWiki uses a command like this for thumbnailing (enabled $wgDebugLog to have it printed out there):

convert '-quality' '80' '-background' 'white' '-define' 'jpeg:size=480x321' 'NRCSIL99001_-_Illinois_(4171)(NRCS_Photo_Gallery).jpg' '-thumbnail' '480x321!' '-set' 'comment' 'File source: http://localhost:3080/wiki/Plik:NRCSIL99001_-_Illinois_(4171)(NRCS_Photo_Gallery).jpg' '+set' 'Thumb::URI' '-depth' '8' '-sharpen' '0x0.4' '-rotate' '-0' '-sampling-factor' '2x2,1x1,1x1' 'out.jpg'

These versions are definitely buggy:

  • ImageMagick 6.8.9-9 (default in Ubuntu 16, and also what we're running in production)
  • ImageMagick 6.9.0-0 (random one I had installed locally)

ImageMagick 6.8.0 and ImageMagick 7.0.0 both thumbnail the files correctly, so this was broken in ImageMagick somewhere between 6.8 and 6.9, and fixed between 6.9 and 7.0.


I supposed the solution is to upgrade or downgrade the version we're using. I'll try out some more versions and pinpoint when it was broken/fixed. @Gilles Is Thumbor using the older, or the newer non-broken version? I have no idea where to check this.

6.8.4 is the first buggy version:


7.0.0 is the first fixed version:

Dereckson added a subscriber: Dereckson.EditedNov 10 2016, 2:59 PM

To summarize a discussion on wikimedia operations IRC channel:

  • ImageMagick 7 API evolved, A porting guide is available.
  • A locally maintained package would offer more coherence when we use several OSes to deploy the same version everywhere, at the cost to require package maintenance
  • To fix this issue, pending ImageMagick 7 testing, per T141739#2785739, the more careful path should be to use 6.8.3, a version we should recommend on documentation, and deploy on wmf cluster
matmarex added a comment.EditedNov 10 2016, 3:52 PM

Just for fun, here are the thumbnails (120x120px square) that MediaWiki produces for all of the problematic files reported here and on related tasks (except the ones that were since deleted) with each of the four interesting ImageMagick versions (6.7.7 which we used previously and to which we could roll back; 6.8.3 which is the last good one; 6.8.4 which is broken; and 7.0.3 which is the latest).

Jdforrester-WMF renamed this task from Issues with displaying thumbnails for CMYK JPG images to Issues with displaying thumbnails for CMYK JPG images due to buggy version of ImageMagick.Nov 10 2016, 4:12 PM
Jdforrester-WMF triaged this task as Medium priority.
Jdforrester-WMF added a subscriber: Jdforrester-WMF.
matmarex renamed this task from Issues with displaying thumbnails for CMYK JPG images due to buggy version of ImageMagick to Issues with displaying thumbnails for CMYK JPG images due to buggy version of ImageMagick (black horizontal stripes, black color missing).Nov 10 2016, 5:05 PM

If we can pinpoint this to a specific commit in 7, which is backportable to from jessie, we can work with the Debian maintainer to backport the fix into a Debian point release.

It appears that the vast majority of commits to ImageMagick have no commit message at all… I suppose I could try to bisect it to find the fix. I wonder how long it takes to compile.

I tried to bisect between IM 6.8.3-10 and 6.8.4-0 first, hoping I could find the problematic patch and revert it, but the commit introducing this issue is d9f6715841cdad02c37efd64308e85282da580b8, which is somewhat unhelpful for understanding the problem, and doesn't clearly revert on 6.9.6-2.

Looks like this is a part of a chain of commits that leads to 524dfaa16d659a2d5c2ac46c4d4c45f5e61ae23c, adding a release note: "The -blur, -guassian-blur, and -sharpen are now convenience methods for -morphology convolve." So perhaps it was just code cleanup… maybe I'll try a bit harder to revert it.

It also seems that removing '-sharpen' '0x0.4' from the command line clears up the issue. So the minimal test case is:

convert -colorspace CMYK logo: '-sharpen' '0x0.4' logo.jpg

Just sharpening, or just working with CMYK images, is fine:

convert logo: '-sharpen' '0x0.4'
convert -colorspace CMYK logo:

IM 7 seems to have diverged from IM 6 in 2011 (!), and from what I've seen in the history so far they like to rewrite something important in every other point release, so I doubt that the fix will be backportable even if I do manage to find one. Honestly, I suspect the commits which broke this were just never forwardported to IM 7, since they were done in 2013.

$ git merge-base master ImageMagick-6

$ git show --stat 1b5421762851bf1649cefc8f5afc4e684f6b68a2 | grep Date
Date:   Sat Jul 9 16:04:35 2011 +0000

$ git log --oneline 1b5421762851bf1649cefc8f5afc4e684f6b68a2..master | wc -l

$ git log --oneline 1b5421762851bf1649cefc8f5afc4e684f6b68a2..ImageMagick-6 | wc -l

For future reference, because there are no tags for older releases in their Git repo:

  • 6.8.3-10 = 8016577473b0eb6bd88c59e7412ed9291e19d671
  • 6.8.4-0 = d9bbeb02f3613640092aef9e2277d997ed9986c6

By the way, looking for alternatives: -convolve is broken in exactly the same way (syntax is weird, but try the example from -unsharp is broken in a different but also hopeless way, I'm not even going to look since when. So patching/upgrading/downgrading looks like the only solution.

Original-sharpen 0x0.4 -unsharp 0x0.4

@MoritzMuehlenhoff Here's the best patch I can produce:

This change reverts the functions ConvolveImage(), ConvolveImageChannel(), SharpenImage(), SharpenImageChannel() to exactly how they looked in ImageMagick 6.8.3-10, which is the last version not exhibiting the issue. It applies on top of 6.9.6-2, compiles and as far as I can tell works as expected.

Do you think this would be okay?

We should probably report this upstream too. I couldn't find any recent cases on the internet of anyone complaining about sharpening CMYK files with ImageMagick. There are some old cases (including our own tasks), but they indicate the issues were fixed at the time (affecting IM 6.5 and older, mostly).

matmarex updated the task description. (Show Details)Nov 12 2016, 8:24 AM
Josve05a moved this task from Backlog to Reported Upstream on the Upstream board.
Gilles added a comment.EditedNov 14 2016, 9:23 AM

The Thumbor servers are on Debian Jessie, which currently uses IM version 8:

Which means with patches applied in the Debian package.

I see nothing in that would mention this. Perhaps Thumbor just doesn't sharpen the thumbnails?

Gilles added a comment.EditedNov 14 2016, 10:21 AM

It does sharpen, but not using the same method as Mediawiki. It currently uses the Python bindings' unsharp_mask() method. Which should be the same as -unsharp in the IM command line.

Specifically, for thumbnails that match the resize ratio condition that triggers sharpening (not all thumbnails get sharpened), it uses the equivalent of -unsharp 0.0x0.8+1.0+0.0 which using DSSIM I found the be the closest visually to -sharpen 0x0.8 which is what's currently used in production.

Hmm, I found -unsharp when looking for alternatives, but I found it to also produce artifacts (different kind though):

By the way, looking for alternatives: -convolve is broken in exactly the same way (syntax is weird, but try the example from -unsharp is broken in a different but also hopeless way, I'm not even going to look since when. So patching/upgrading/downgrading looks like the only solution.

Original-sharpen 0x0.4 -unsharp 0x0.4

Can you put the original image here (generated with convert -colorspace CMYK logo: logo.jpg) through Thumbor and see if the colors of the result look right? Perhaps this is specific to some version, or some values for -unsharp.

-unsharp was also tried in the past, then swiftly reverted: T26857.

I would advise against subjective judgment regarding visual artifacts. This is what algorithms like DSSIM are for. We look closely at status quo thumbnails all the time, which makes us "used" to the kind of artifacts it generates and a new kind of visual artifact looks more wrong to our eyes than it should. While it might actually preserve more - or as much - visual information.

DSSIM is a much more objective way to compare visual variants of the same image.

I'll generate what you've requested in a sec.

Generated in production.

300px coming from Mediawiki:

300px coming from Thumbor:

The softness looks out of place, but the colors look accurate.

Generating an original-sized image with sharpening applied would be kind of irrelevant, because not only we don't currently allow original-sized thumbnails, sharpening is only applied when the resize ratio is greater than a defined minimum. Images whose size is close to the original aren't sharpened.

Um, perhaps you're seeing something else than I do (you never know with these things…), but I'm seeing completely different colors in the processed image than the original. I'm not talking small details :)

Yes, I was doing sharpening-only as a "minimal test case" of sorts. Thanks. So I guess just waiting on Thumbor is also an option to solve this.

Um, perhaps you're seeing something else than I do (you never know with these things…), but I'm seeing completely different colors in the processed image than the original. I'm not talking small details :)

In Chromium the colors are identical between the original you've linked to and the thumbor-generated thumbnail, but then again I'm looking at this inside a vmware VM on an external monitor, so yeah, who knows. At least make sure that you're looking at images on the same monitor.

In the 3 images you've provided, the -sharpen one has the same colors and the production Mediawiki example I've provided. And your -unsharp one looks like it has very high contrast, the wizard's suit is almost black instead of blue.

Note that Thumbor, just like Mediawiki currently, leaves color profiles alone when they're not sRGB. That might play a role in having a different experience looking at the same images. And phabricator might mess with that stuff in the embedded previews, make sure to open the actual attachments.

OK, upstream have released ImageMagick 6.9.6-5 with a fix for the issue.

I'll look into a backport to our jessie package currently used on the image scalers, if the patch is self-contained and sane we might be able to get that into the next imagemagick update in jessie.

Mentioned in SAL (#wikimedia-operations) [2016-11-17T07:57:54Z] <moritzm> uploaded imagemagick 8: to carbon (T141739)

Mentioned in SAL (#wikimedia-operations) [2016-11-17T10:20:16Z] <moritzm> upgrading imagemagick on mw1293 (T141739)

Mentioned in SAL (#wikimedia-operations) [2016-11-17T12:50:41Z] <moritzm> upgrading imagemagick on remaining image scalers (T141739)

The backported patch is now deployed on the image scalers, I'll also report this in the Debian bug tracker, ideally we can merge that into the next jessie point release so that we can use unmodified imagemagick builds in the future.

matmarex closed this task as Resolved.Nov 17 2016, 3:33 PM

@Gilles @MoritzMuehlenhoff Thank you! Everything looks fine on the wikis now. I purged the files linked on this task that were still affected, and they thumbnail correctly now.

Dzahn added a subscriber: Dzahn.Dec 2 2016, 6:31 AM

imagescalers and thumbors upgraded from imagemagick 8: to imagemagick 8:

security upgrade and our patch here for this on top of it again

Dzahn added a comment.Dec 2 2016, 6:41 AM

2[neodymium:~/debdeploy] $ cat 2016-12-01-imagemagick.yaml
3source: imagemagick
4comment: DSA-3726-1 security update
5update_type: tool
7 precise:
8 jessie: 8:
9 trusty:
13[neodymium:~/debdeploy] $ sudo debdeploy -u 2016-12-01-imagemagick.yaml -s imagescaler-eqiad status-deploy
15 Updated packages:
16 imagemagick-common: 8: -> 8:
17 imagemagick: 8: -> 8:
18 libmagickwand-6.q16-2: 8: -> 8:
19 imagemagick-6.q16: 8: -> 8:
20 libmagickcore-6.q16-2: 8: -> 8:
22 Updated packages:
23 imagemagick-common: 8: -> 8:
24 imagemagick: 8: -> 8:
25 libmagickwand-6.q16-2: 8: -> 8:
26 imagemagick-6.q16: 8: -> 8:
27 libmagickcore-6.q16-2: 8: -> 8:
29 Updated packages:
30 imagemagick-common: 8: -> 8:
31 libmagickwand-6.q16-2: 8: -> 8:
32 libmagickcore-6.q16-2: 8: -> 8:
34 Updated packages:
35 imagemagick-common: 8: -> 8:
36 imagemagick: 8: -> 8:
37 libmagickwand-6.q16-2: 8: -> 8:
38 imagemagick-6.q16: 8: -> 8:
39 libmagickcore-6.q16-2: 8: -> 8:
41 Updated packages:
42 imagemagick-common: 8: -> 8:
43 imagemagick: 8: -> 8:
44 libmagickwand-6.q16-2: 8: -> 8:
45 imagemagick-6.q16: 8: -> 8:
46 libmagickcore-6.q16-2: 8: -> 8:
48 Updated packages:
49 imagemagick-common: 8: -> 8:
50 imagemagick: 8: -> 8:
51 libmagickwand-6.q16-2: 8: -> 8:
52 imagemagick-6.q16: 8: -> 8:
53 libmagickcore-6.q16-2: 8: -> 8:
56Deployment summary:
57Number of hosts in this deployment run: 6
58No packages were added
59No packages were removed
60Updated packages:
61imagemagick-6.q16: 8: -> 8: on 5 hosts
62imagemagick-common: 8: -> 8: on 1 hosts
63imagemagick-common: 8: -> 8: on 5 hosts
64imagemagick: 8: -> 8: on 5 hosts
65libmagickcore-6.q16-2: 8: -> 8: on 1 hosts
66libmagickcore-6.q16-2: 8: -> 8: on 5 hosts
67libmagickwand-6.q16-2: 8: -> 8: on 1 hosts
68libmagickwand-6.q16-2: 8: -> 8: on 5 hosts
70No restarts are needed
72Error summary:
73No errors found
77[neodymium:~/debdeploy] $ sudo debdeploy -u 2016-12-01-imagemagick.yaml -s imagescaler-codfw status-deploy
79 Updated packages:
80 imagemagick-common: 8: -> 8:
81 imagemagick: 8: -> 8:
82 libmagickwand-6.q16-2: 8: -> 8:
83 imagemagick-6.q16: 8: -> 8:
84 libmagickcore-6.q16-2: 8: -> 8:
86 Updated packages:
87 imagemagick-common: 8: -> 8:
88 imagemagick: 8: -> 8:
89 libmagickwand-6.q16-2: 8: -> 8:
90 imagemagick-6.q16: 8: -> 8:
91 libmagickcore-6.q16-2: 8: -> 8:
93 Updated packages:
94 imagemagick-common: 8: -> 8:
95 libmagickwand-6.q16-2: 8: -> 8:
96 libmagickcore-6.q16-2: 8: -> 8:
98 Updated packages:
99 imagemagick-common: 8: -> 8:
100 imagemagick: 8: -> 8:
101 libmagickwand-6.q16-2: 8: -> 8:
102 imagemagick-6.q16: 8: -> 8:
103 libmagickcore-6.q16-2: 8: -> 8:
105 Updated packages:
106 imagemagick-common: 8: -> 8:
107 imagemagick: 8: -> 8:
108 libmagickwand-6.q16-2: 8: -> 8:
109 imagemagick-6.q16: 8: -> 8:
110 libmagickcore-6.q16-2: 8: -> 8:
112 Updated packages:
113 imagemagick-common: 8: -> 8:
114 imagemagick: 8: -> 8:
115 libmagickwand-6.q16-2: 8: -> 8:
116 imagemagick-6.q16: 8: -> 8:
117 libmagickcore-6.q16-2: 8: -> 8:
119 Updated packages:
120 imagemagick-common: 8: -> 8:
121 imagemagick: 8: -> 8:
122 libmagickwand-6.q16-2: 8: -> 8:
123 imagemagick-6.q16: 8: -> 8:
124 libmagickcore-6.q16-2: 8: -> 8:
126 Updated packages:
127 imagemagick-common: 8: -> 8:
128 imagemagick: 8: -> 8:
129 libmagickwand-6.q16-2: 8: -> 8:
130 imagemagick-6.q16: 8: -> 8:
131 libmagickcore-6.q16-2: 8: -> 8:
134Deployment summary:
135Number of hosts in this deployment run: 8
136No packages were added
137No packages were removed
138Updated packages:
139imagemagick-6.q16: 8: -> 8: on 7 hosts
140imagemagick-common: 8: -> 8: on 7 hosts
141imagemagick-common: 8: -> 8: on 1 hosts
142imagemagick: 8: -> 8: on 7 hosts
143libmagickcore-6.q16-2: 8: -> 8: on 7 hosts
144libmagickcore-6.q16-2: 8: -> 8: on 1 hosts
145libmagickwand-6.q16-2: 8: -> 8: on 7 hosts
146libmagickwand-6.q16-2: 8: -> 8: on 1 hosts
148No restarts are needed
150Error summary:
151No errors found
154neodymium:~/debdeploy] $ sudo debdeploy -u 2016-12-01-imagemagick.yaml -s thumbor status-deploy
156 No change
158 No change
161Deployment summary:
162Number of hosts in this deployment run: 2
163No packages were added
164No packages were removed
165No packages were updated
167No restarts are needed
169Error summary:
170No errors found
174imagemagick (8: jessie-security; urgency=medium
176 * Fix convert -sharpen with CYMK images (Bug: T141739)
178 -- Daniel Zahn <> Thu, 1 Dec 2016 18:33:33 -0800
180imagemagick (8: jessie-security; urgency=medium
182 * Fix CVE-2016-7799: global buffer overflow. (Closes: #840437).
183 * Fix CVE-2016-7906: use after free. (Closes: #840435).
184 * Fix a TIFF file buffer overflow. (Closes: #845195).
185 * Check return of fputc during TIFF file writing.
186 (Closes: #845196).
187 * Prevent buffer overflow by checking image extend
188 for TIFF (Closes: #845198).
189 * Avoid a out of bound read in VIFF file handler.
190 (Closes: #845212 and LP: #1545183).
191 * Avoid a DOS by not allowing too deep nested exception.
192 (Closes: #845213).
193 * Better check for buffer overflow in TIFF files
194 handling. (Closes: #845202).
195 * Fix CVE-2016-8677: memory allocate failure in AcquireQuantumPixels
196 (Closes: #845206).
197 * Prevent fault in MSL interpreter. (Closes: #845242).
198 * Prevent heap buffer overflow in heap-buffer-overflow in IsPixelGray
199 (Closes: #845242)
200 * Fix null pointer dereference in TIFF file handling.
201 (Closes: #845243).
202 * Added check for invalid number of frames in mat file
203 (Closes: #845244).
204 * Fix an out of bound read in mat file due to insuffisant allocation.
205 (Closes: #845246).
206 * Fix CVE-2016-8862: memory allocation failure in AcquireMagickMemory
207 (Closes: #845634).
210root@carbon:~# reprepro ls imagemagick
211imagemagick | 8: | jessie-wikimedia | amd64, source
215Format: 1.8
216Date: Thu, 1 Dec 2016 18:33:33 -0800
217Source: imagemagick
218Binary: imagemagick-common imagemagick-doc libmagickcore-6-headers libmagickwand-6-headers libmagick++-6-headers imagemagick libimage-magick-perl libmagickcore-6-arch-config imagemagick-6.q16 libmagickcore-6.q16-2 libmagickcore-6.q16-2-extra libmagickcore-6.q16-dev libmagickwand-6.q16-2 libmagickwand-6.q16-dev libmagick++-6.q16-5 libmagick++-6.q16-dev imagemagick-dbg libimage-magick-q16-perl perlmagick libmagickcore-dev libmagickwand-dev libmagick++-dev
219Architecture: source all amd64
220Version: 8:
221Distribution: jessie-wikimedia
222Urgency: medium
223Maintainer: ImageMagick Packaging Team <>
224Changed-By: Daniel Zahn <>
226 imagemagick - image manipulation programs -- binaries
227 imagemagick-6.q16 - image manipulation programs -- quantum depth Q16
228 imagemagick-common - image manipulation programs -- infrastructure
229 imagemagick-dbg - debugging symbols for ImageMagick
230 imagemagick-doc - document files of ImageMagick
231 libimage-magick-perl - Perl interface to the ImageMagick graphics routines
232 libimage-magick-q16-perl - Perl interface to the ImageMagick graphics routines -- Q16 versio
233 libmagick++-6-headers - object-oriented C++ interface to ImageMagick - header files
234 libmagick++-6.q16-5 - object-oriented C++ interface to ImageMagick
235 libmagick++-6.q16-dev - object-oriented C++ interface to ImageMagick - development files
236 libmagick++-dev - object-oriented C++ interface to ImageMagick
237 libmagickcore-6-arch-config - low-level image manipulation library - architecture header files
238 libmagickcore-6-headers - low-level image manipulation library - header files
239 libmagickcore-6.q16-2 - low-level image manipulation library -- quantum depth Q16
240 libmagickcore-6.q16-2-extra - low-level image manipulation library - extra codecs (Q16)
241 libmagickcore-6.q16-dev - low-level image manipulation library - development files (Q16)
242 libmagickcore-dev - low-level image manipulation library -- transition package
243 libmagickwand-6-headers - image manipulation library - headers files
244 libmagickwand-6.q16-2 - image manipulation library
245 libmagickwand-6.q16-dev - image manipulation library - development files
246 libmagickwand-dev - image manipulation library - transition for development files
247 perlmagick - Perl interface to ImageMagick -- transition package
249 imagemagick (8: jessie-security; urgency=medium
250 .
251 * Fix convert -sharpen with CYMK images (Bug: T141739)
253 273388417e80f2e2753c09aa9b35496d89e9c866 3379 imagemagick_6.8.9.9-5+deb8u6+wmf1.dsc

I've filed to get this integrated into Debian jessie as well.

The update has been accepted by the Debian stable release managers, so starting with the next jessie update we can use the stock Debian packages again.

Have we reverted, or do we have a regression on our hand?

Full versionCommons thumbnail
Josve05a reopened this task as Open.Apr 28 2017, 4:50 AM

No, that seems like a new/unrelated issue which simply exhibits a similar visual distortion. We're still using the same imagemagick version and the original bug is still fixed, so let's make this a new Phab task.

Aklapper closed this task as Resolved.Apr 28 2017, 1:06 PM

Restoring previous task status per previous comment.
@Josve05a: Please create a separate task - thanks!

I don't actually see this problem on that file. Looks like it was uploaded before this task was resolved, so it might have just been an old broken thumbnail that no one purged.