At first glance this looks like an issue with the new thumbnailing backend. I'll look into this ASAP.

Change 361689 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Switch back to image scalers

https://gerrit.wikimedia.org/r/361689

Change 361689 merged by Filippo Giunchedi:
[operations/puppet@production] Switch back to image scalers

https://gerrit.wikimedia.org/r/361689

Mentioned in SAL (#wikimedia-operations) [2017-06-27T16:23:22Z] <godog> revert back to imagescalers for thumbs - T168949

Purging affected files should fix them. We've just purged https://commons.wikimedia.org/wiki/File:Green_and_golden_Butterfly.jpg

Change 361709 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 0.1.41

https://gerrit.wikimedia.org/r/361709

Change 361709 merged by Filippo Giunchedi:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 0.1.41

https://gerrit.wikimedia.org/r/361709

Change 361711 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Switch Thumbor back on

https://gerrit.wikimedia.org/r/361711

Change 361711 merged by Filippo Giunchedi:
[operations/puppet@production] Switch Thumbor back on

https://gerrit.wikimedia.org/r/361711

Mentioned in SAL (#wikimedia-operations) [2017-06-27T18:41:34Z] <godog> switch thumbor back on with a fix for T168949

We'll try to purge all images that were potentially affected during the breakage window. But it's probably going to take a few days to write a dedicated cleanup script and run it. In the meantime, just purge any affected file you come across.

Only JPGs should be affected, so we can reduce candidate images in the cleanup window to just those. Then we can query the Mediawiki API to filter out potential candidates. The following command (insert proper domain and file name where appropriate) echoes "true" if the EXIF rotation of the image is 90 or 270:

curl -s 'https://en.wikipedia.org/w/api.php?action=query&titles=File:Green_and_golden_Butterfly_copy.jpg&prop=imageinfo&iiprop=metadata&format=json' | jq '.query.pages | values[].imageinfo[0].metadata | map(select(.name == "Orientation"))[0].value >= 5'

The -e option of jq can be used if it's desirable to have the exit status of the command match the output (exit status 1 for false, 0 for true).

With that I think we can reduce the 200s in the nginx logs during the window to only the files that were really affected.

@fgiunchedi can you put the deduped wiki + filename list gathered from all thumbor hosts somewhere?

@Gilles nice! I've left a tar of all (not only jpegs) files from nginx 200s on thumbor machines at terbium.eqiad.wmnet:~filippo/thumbs_to_purge_T168949.tar.gz

I've purged affected images on all wikis except Commons, which is still ongoing. It will probably take a few hours to complete.

Hmmm the rate limit is very low, even sleeping 1s after each purge isn't enough. I think I need set up an exception for that script or something. Or I guess write a maintenance script instead of doing this through the API...

In T168949#3389821, @Gilles wrote:

When working on T169072: Due to missing fonts on Thumbor hosts, PNG thumbnail of SVG file showing a Devanagari character only displays a Unicode number box I realized that I wasn't displaying errors when the rate limiting was hit, which means that I might have missed some purges due to that throttling...

@fgiunchedi could you re-generate this list in the format you've made for the newest one? (single file, space-separated wiki name and title).

For sure, it is on terbium:~filippo/thumbs_purgelist. The list is all files requested, regardless of extension, so you'll need to filter for jpe?g

I am getting "Duplicate headers received from server" and "Error code: ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION" at https://upload.wikimedia.org/wikipedia/commons/thumb/b/b1/Bundesarchiv_Bild_183-14581-0003%2C_Dresden%2C_Omnibus_%22Ikarus_30%22.jpg/180px-Bundesarchiv_Bild_183-14581-0003%2C_Dresden%2C_Omnibus_%22Ikarus_30%22.jpg when trying to display the 180px thumbnail of https://commons.wikimedia.org/wiki/File:Bundesarchiv_Bild_183-14581-0003,_Dresden,_Omnibus_%22Ikarus_30%22.jpg as my most recent upload at https://commons.wikimedia.org/w/index.php?title=Special:ListFiles&offset=20170712204449&user=Jeff+G.&ilshowall=1&limit=1 . This is after purging, the not displayed effect persists from before the purging. :(

@Jeff_G please create a separate task for this issue with more details (which browser, browser version, a screenshot of the error, etc.). It's unrelated to this task.

In T168949#3434184, @Gilles wrote:

@Jeff_G please create a separate task for this issue with more details (which browser, browser version, a screenshot of the error, etc.). It's unrelated to this task.

Thanks, done at T170605.

(also mentioned in https://commons.wikimedia.org/wiki/Commons:Village_pump#Problems_with_thumbnail_generation.3F )

In T168949#3515066, @Aklapper wrote:

(also mentioned in https://commons.wikimedia.org/wiki/Commons:Village_pump#Problems_with_thumbnail_generation.3F )

Unrelated, should be a different task. If I open those 3 originals, they're visually cut off just the same as the thumbnails: https://upload.wikimedia.org/wikipedia/commons/c/cd/Congress_Building.jpg

This would suggest an issue at the intake or in the storage of originals.