Page MenuHomePhabricator
Create Task
Maniphest T226707

cwebp chokes on YCCK JPGs
Closed, ResolvedPublic
Actions

Description

I tried uploading the same file to beta and it's spinning for a long time before failing with a 504.

Trying to run a cwebp command on thumbor2001 I'm getting an instantaneous error:

gilles@thumbor2001:~$ cwebp 113px-Constitution_of_India.jpg -metadata all -q 50 -quiet -o -
Unsupported color conversion request
Error! Could not process file 113px-Constitution_of_India.jpg
Error! Cannot read input picture file '113px-Constitution_of_India.jpg'

I don't see, however, what might be causing the hanging.

Details

SubjectRepoBranchLines +/-
Upgrade to 2.6operations/debs/python-thumbor-wikimediamaster+16 -6
Version bumpoperations/software/thumbor-pluginsmaster+1 -1
Serve JPG when WEBP conversion failsoperations/software/thumbor-pluginsmaster+9 -5
Customize query in gerrit

Related Objects

Event Timeline

• Gilles created this task.Jun 27 2019, 11:42 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2019, 11:42 AM
• Gilles added a comment.Jun 27 2019, 11:53 AM

It seems like cwebp doesn't like the YCCK color space that file uses:

Color Transform                 : YCCK

One workaround might be to convert the file to sRGB first before invoking cwebp

• Gilles renamed this task from Constitution_of_India.jpg webp variant stuck in poolcounter per-original throttle to cwebp chokes on YCCK JPGs.Jun 27 2019, 11:53 AM
• Gilles added a comment.Jun 27 2019, 11:57 AM

Alternatively, we could just stick to the JPG we've already generated when a webp is requested and the cwebp command fails. It will probably make for simpler code and foolproof for other unforeseen situations where cwebp fails. What's unclear still is why things are hanging in case of cwebp command error.

gerritbot added a comment.Jun 27 2019, 12:01 PM

Change 519379 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/software/thumbor-plugins@master] Serve JPG when WEBP conversion fails

https://gerrit.wikimedia.org/r/519379

gerritbot added a project: Patch-For-Review.Jun 27 2019, 12:01 PM
• Gilles mentioned this in rTHMBREXTe7d48d925c84: Serve JPG when WEBP conversion fails.Jun 27 2019, 12:27 PM
• Gilles mentioned this in rTHMBREXT6975dae96666: Serve JPG when WEBP conversion fails.Jun 27 2019, 10:01 PM
gerritbot added a comment.Jun 29 2019, 2:01 PM

Change 519379 merged by Gilles:
[operations/software/thumbor-plugins@master] Serve JPG when WEBP conversion fails

https://gerrit.wikimedia.org/r/519379

gerritbot added a comment.Jun 29 2019, 2:05 PM

Change 519755 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/software/thumbor-plugins@master] Version bump

https://gerrit.wikimedia.org/r/519755

• Gilles mentioned this in rTHMBREXTf0e5319c7b84: Version bump.Jun 29 2019, 2:09 PM
gerritbot added a comment.Jun 29 2019, 2:40 PM

Change 519755 merged by Gilles:
[operations/software/thumbor-plugins@master] Version bump

https://gerrit.wikimedia.org/r/519755

Maintenance_bot removed a project: Patch-For-Review.Jun 29 2019, 3:10 PM
Krinkle claimed this task.Jul 1 2019, 7:53 PM
Krinkle subscribed.

To ask Filippo whether it has been deployed and if not, whether they can do that for us.

Krinkle moved this task from Inbox, needs triage to Doing (old) on the Performance-Team board.Jul 1 2019, 7:54 PM
• Gilles added a comment.Jul 3 2019, 11:25 AM

This needs a thumbor-plugins Debian package update with the latest code + deployment.

Krinkle reassigned this task from Krinkle to • Gilles.Jul 5 2019, 6:39 PM
Krinkle added a subscriber: jijiki.

Yeah, forgot about that. Before I realised that, I asked Filippo on Monday and he said it (indeed..) hadn't been deployed yet. Reminder to self: Should have asked @jijiki instead, who does SRE for Thumbor now.

Krinkle moved this task from Doing (old) to To-do: Goals prioritized current Quarter on the Performance-Team board.Aug 6 2019, 1:11 AM
• Gilles added a comment.Aug 19 2019, 4:17 PM

@jijiki what's the status of this? Can we deploy it this week if it hasn't been deployed yet?

• Gilles mentioned this in T228467: thumbtime/seek thumbnailing broken with floating point offsets.Aug 19 2019, 4:52 PM
• Gilles triaged this task as High priority.Aug 20 2019, 2:22 PM
gerritbot added a comment.Aug 20 2019, 2:28 PM

Change 531204 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.6

https://gerrit.wikimedia.org/r/531204

gerritbot added a project: Patch-For-Review.Aug 20 2019, 2:28 PM
• Gilles moved this task from To-do: Goals prioritized current Quarter to Blocked (old) on the Performance-Team board.Sep 30 2019, 8:38 PM
• Gilles added a comment.Oct 1 2019, 7:59 AM

@jijiki do you have some spare time in the next couple of weeks to deploy this?

gerritbot added a comment.Nov 21 2019, 12:09 PM

Change 531204 merged by Effie Mouzeli:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.6

https://gerrit.wikimedia.org/r/531204

Maintenance_bot removed a project: Patch-For-Review.Nov 21 2019, 12:10 PM
• Gilles added a comment.Jan 6 2020, 1:12 PM

@jijiki as far as I can tell, this was never roll-deployed?

gilles@thumbor2001:~$ dpkg -l python-thumbor-wikimedia
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                               Version                Architecture           Description
+++-==================================-======================-======================-=========================================================================
ii  python-thumbor-wikimedia           2.4-1+deb9u1           all                    Thumbor wikimedia extensions
• Gilles lowered the priority of this task from High to Medium.Jan 6 2020, 1:15 PM
• Gilles lowered the priority of this task from Medium to Low.
Krinkle unsubscribed.Jan 6 2020, 8:13 PM
• Gilles added a comment.Apr 8 2020, 8:51 AM

@jijiki can we schedule a day and time to deploy this update?

• Gilles changed the task status from Open to Stalled.May 4 2020, 11:46 AM

Blocked on deployment by SRE

• Gilles added a project: SRE.May 11 2020, 1:46 PM
• ema changed the task status from Stalled to Open.May 12 2020, 10:24 AM
Stashbot added a comment.May 12 2020, 10:30 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-12T10:30:07Z] <ema> rolling thumbor upgrade to 2.6-1+deb10u1 T226707

• ema subscribed.May 12 2020, 10:48 AM

If I understand the issue correctly, it looks fixed to me now. The following is now returned as CT:image/jpeg:

10:45:56 ema@cp1076.eqiad.wmnet:~
$ curl -i https://swift.discovery.wmnet/wikipedia/commons/thumb/4/4d/Constitution_of_India.jpg/120px-Constitution_of_India.jpg.webp 
HTTP/2 200 
date: Tue, 12 May 2020 10:46:05 GMT
content-type: image/jpeg
content-length: 29753
accept-ranges: bytes
last-modified: Tue, 12 May 2020 10:45:22 GMT
etag: 25728b156d2589b461b4503dbf98cc79
x-timestamp: 1589280321.64762
access-control-allow-origin: *
x-trans-id: tx7a013b5bb1994bf5baded-005eba7e6d
• Gilles closed this task as Resolved.May 12 2020, 11:18 AM

YCCK images requested as webp now fall back correctly to jpeg

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL