Page MenuHomePhabricator

Proper thumbnails of portrait photos not being generated; serious display issues
Closed, ResolvedPublic

Description

Thumbnails of many images on Commons and English Wikipedia are appearing grainy, because the appropriately sized copies are not being generated. MediaWiki then upscales a low-res version, with hideous results.

Affected images appear to mostly be large JPEGs with a portrait orientation (long Y axis).

See these pages for examples:
https://commons.wikimedia.org/wiki/Commons:Graphics_village_pump
https://test.wikipedia.org/wiki/User:Guanaco/sandbox

Also affects local uploads:
https://en.wikipedia.org/wiki/File:Green_and_golden_Butterfly_copy.jpg

Event Timeline

Guanaco created this task.Jun 27 2017, 1:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2017, 1:24 PM
Guanaco triaged this task as High priority.Jun 27 2017, 1:24 PM
Guanaco updated the task description. (Show Details)
Aklapper edited projects, added Thumbor; removed MediaWiki-Cache.Jun 27 2017, 2:44 PM
Aklapper added a subscriber: Gilles.

At first glance this looks like an issue with the new thumbnailing backend. I'll look into this ASAP.

Gilles raised the priority of this task from High to Unbreak Now!.
Gilles added a subscriber: fgiunchedi.
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptJun 27 2017, 4:16 PM

Change 361689 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Switch back to image scalers

https://gerrit.wikimedia.org/r/361689

Change 361689 merged by Filippo Giunchedi:
[operations/puppet@production] Switch back to image scalers

https://gerrit.wikimedia.org/r/361689

Mentioned in SAL (#wikimedia-operations) [2017-06-27T16:23:22Z] <godog> revert back to imagescalers for thumbs - T168949

Gilles lowered the priority of this task from Unbreak Now! to Normal.Jun 27 2017, 4:31 PM

I'll work on a fix for Thumbor and we'll redeploy it once it's fixed.

Change 361709 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 0.1.41

https://gerrit.wikimedia.org/r/361709

Change 361709 merged by Filippo Giunchedi:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 0.1.41

https://gerrit.wikimedia.org/r/361709

Change 361711 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Switch Thumbor back on

https://gerrit.wikimedia.org/r/361711

Change 361711 merged by Filippo Giunchedi:
[operations/puppet@production] Switch Thumbor back on

https://gerrit.wikimedia.org/r/361711

Mentioned in SAL (#wikimedia-operations) [2017-06-27T18:41:34Z] <godog> switch thumbor back on with a fix for T168949

We'll try to purge all images that were potentially affected during the breakage window. But it's probably going to take a few days to write a dedicated cleanup script and run it. In the meantime, just purge any affected file you come across.

Only JPGs should be affected, so we can reduce candidate images in the cleanup window to just those. Then we can query the Mediawiki API to filter out potential candidates. The following command (insert proper domain and file name where appropriate) echoes "true" if the EXIF rotation of the image is 90 or 270:

curl -s 'https://en.wikipedia.org/w/api.php?action=query&titles=File:Green_and_golden_Butterfly_copy.jpg&prop=imageinfo&iiprop=metadata&format=json' | jq '.query.pages | values[].imageinfo[0].metadata | map(select(.name == "Orientation"))[0].value >= 5'

The -e option of jq can be used if it's desirable to have the exit status of the command match the output (exit status 1 for false, 0 for true).

With that I think we can reduce the 200s in the nginx logs during the window to only the files that were really affected.

@fgiunchedi can you put the deduped wiki + filename list gathered from all thumbor hosts somewhere?

@Gilles nice! I've left a tar of all (not only jpegs) files from nginx 200s on thumbor machines at terbium.eqiad.wmnet:~filippo/thumbs_to_purge_T168949.tar.gz

I've purged affected images on all wikis except Commons, which is still ongoing. It will probably take a few hours to complete.

Gilles closed this task as Resolved.Jun 29 2017, 8:02 AM

All files from the list have been purged. Everything should be fixed except older versions of the files that might have had a new thumbnail created (probably a very small amount of thumbnails affected), because of purging's shortcomings. Which is the subject of T169178: Purging files doesn't purge the thumbnails of older versions

Gilles reopened this task as Open.Jun 29 2017, 8:49 AM

When working on T169072: Due to missing fonts on Thumbor hosts, PNG thumbnail of SVG file showing a Devanagari character only displays a Unicode number box I realized that I wasn't displaying errors when the rate limiting was hit, which means that I might have missed some purges due to that throttling...

@fgiunchedi could you re-generate this list in the format you've made for the newest one? (single file, space-separated wiki name and title).

Hmmm the rate limit is very low, even sleeping 1s after each purge isn't enough. I think I need set up an exception for that script or something. Or I guess write a maintenance script instead of doing this through the API...

Gilles changed the task status from Open to Stalled.Jun 29 2017, 10:35 AM
Gilles raised the priority of this task from Normal to High.
Gilles removed a project: Patch-For-Review.

When working on T169072: Due to missing fonts on Thumbor hosts, PNG thumbnail of SVG file showing a Devanagari character only displays a Unicode number box I realized that I wasn't displaying errors when the rate limiting was hit, which means that I might have missed some purges due to that throttling...
@fgiunchedi could you re-generate this list in the format you've made for the newest one? (single file, space-separated wiki name and title).

For sure, it is on terbium:~filippo/thumbs_purgelist. The list is all files requested, regardless of extension, so you'll need to filter for jpe?g

Jeff_G added a subscriber: Jeff_G.Jul 12 2017, 11:21 PM

@Jeff_G please create a separate task for this issue with more details (which browser, browser version, a screenshot of the error, etc.). It's unrelated to this task.

@Jeff_G please create a separate task for this issue with more details (which browser, browser version, a screenshot of the error, etc.). It's unrelated to this task.

Thanks, done at T170605.

Gilles changed the task status from Stalled to Open.Aug 7 2017, 8:26 PM

Unrelated, should be a different task. If I open those 3 originals, they're visually cut off just the same as the thumbnails: https://upload.wikimedia.org/wikipedia/commons/c/cd/Congress_Building.jpg

This would suggest an issue at the intake or in the storage of originals.

Gilles closed this task as Resolved.Sep 4 2017, 1:09 PM

All affected files have been purged.