Can be seen on all tumbnails of https://commons.wikimedia.org/wiki/File:Von_Bach_bis_Tango_(ZMF_2016)_jm61021.jpg

1. Add new image (with this issue) to random Wikipedia article
2. A "new thumb size" of that file is generated
3. Note lines in that version of the file in preview

From server admin log
13:54 moritzm: depooling image scaler mw1298 for some local tests with huge SVGs
12:26 moritzm: depooling image scalers mw2086-mw2089 for reimaging with jessie

I've added Wikimedia-Incident as this is pretty bad.

@MoritzMuehlenhoff @Gilles help asap please. What can we revert to get us back to a good state?

After a fix has been made/patch reverted, we will need to purge/reparse all thumbnails generated during this time which may be "corrupted".

@Bawolff or @brion can either of you help out here, per chance?

@Dereckson in T142638#2542157, depooling a scaler shouldn't matter, and mw2086-mw2089 supposedly are dormant.

I expect the current scalers are mw129[3-8].eqiad.wmnet, which are jessie since 2016-07-04

Mentioned in SAL [2016-08-10T23:17:08Z] <reedy@tin> rebuilt wikiversions.php and synchronized wikiversions files: Revert to .13 to attempt to fix T142638

OK, the rollback looks to have fixed this. Meh.

In T142638#2542184, @Josve05a wrote:

After a fix has been made/patch reverted, we will need to purge/reparse all thumbnails generated during this time which may be "corrupted".

Someone: How can we generate that list?

Mentioned in SAL [2016-08-10T23:40:50Z] <reedy@tin> Synchronized php-1.28.0-wmf.14/extensions/VipsScaler: Remove old broken config causing T142638 (duration: 00m 50s)

Update from IRC:

• We rolled back the train (wmf.14) to wmf.13, which fixed the issue from getting worse.
• We're running a maintenance script, PurgeChangedFiles, to delete the bad thumbnails.
• We found that this happened because MediaWiki-extensions-VipsScaler (which handles JPEG scaling) over-rides configuration for image scaling with its own very poor defaults. In the new train of MediaWiki we changed the code that used that config, which meant it merged the config very poorly. This is why thumbnailing of JPEGs was broken but other types were fine.
• We've removed the defaults from VipsScaler and showed that that worked on test. Now rolling the train back out.

Mentioned in SAL [2016-08-10T23:46:37Z] <reedy@tin> rebuilt wikiversions.php and synchronized wikiversions files: Reinstate .14 as T142638 is fixed

Mentioned in SAL [2016-08-10T23:59:03Z] <AaronSchulz> Running purgeChangedFiles.php on all wikis on a terbium screen (T142638)

In T142638#2542376, @Jdforrester-WMF wrote:

• We're running a maintenance script, PurgeChangedFiles, to delete the bad thumbnails.

Which will only affect files that have been changed, not ones that have been manually purged or just did not have a thumbnail in the request size. I don't think there is a way to identify those though.

https://wikitech.wikimedia.org/wiki/Incident_documentation/20160810-MediaWiki has been written as a result of this outage

In T142638#2542591, @Tgr wrote:

In T142638#2542376, @Jdforrester-WMF wrote:

• We're running a maintenance script, PurgeChangedFiles, to delete the bad thumbnails.

Which will only affect files that have been changed, not ones that have been manually purged or just did not have a thumbnail in the request size. I don't think there is a way to identify those though.

Is there a way from Swift?

Other than that, page_touched value between some dates, and NS = NS_FILE?

If no thumbnail in the request size, aren't they just served the original anyway?

I don't know if we log things at the swift level that would let us know which images were pushed to swift during a certain period. @fgiunchedi might know. He's on vacation this week and back on Monday.

Varnish logs might be another place to look, namely purge requests and varnish backend misses. But I don't know if we keep a durable record of those. @BBlack might know

In T142638#2543570, @Gilles wrote:

I don't know if we log things at the swift level that would let us know which images were pushed to swift during a certain period. @fgiunchedi might know. He's on vacation this week and back on Monday.

we keep swift access logs on ms-fe machines, though only for 5 days given their size :(