@Yann Also, can you identify when the problem started? Commons is still on the version of MediaWiki from last week (except for a brief period yesterday when we tried to roll forward but had to rollback, see: T183966)

OK, I restored the broken file. @greg I have seen hundreds of broken files in the last weeks, while I used to see one in a week previously (compared to last year).
I can't say when it started, but there is a difference of several order of magnitude. These files are usually speedy deleted.

Adding the Multimedia team for their diagnosis of this issue (as they own many of the parts here).

For https://upload.wikimedia.org/wikipedia/commons/7/74/Valentin_Guichaux_posant_%C3%A0_cot%C3%A9_de_son_Mao.jpg :

$:acko\> rpm -q jpeginfo
jpeginfo-1.6.1-8.fc27.x86_64
$:acko\> jpeginfo -c Valentin_Guichaux_posant_à_coté_de_son_Mao.jpg 
Valentin_Guichaux_posant_à_coté_de_son_Mao.jpg 6000 x 4000 24bit Exif  N 5242880  Premature end of JPEG file  [WARNING]

More files:

• https://commons.wikimedia.org/wiki/File:Stuart_-_English_National_Opera_Headshot.jpg
• https://commons.wikimedia.org/wiki/File:Atindras.jpg
• https://commons.wikimedia.org/wiki/File:Ngh%E1%BB%87_s%C4%A9_V%C3%A2n_H%C3%A0.jpg
• https://commons.wikimedia.org/wiki/File:Carleton_at_Sevilla_FC.jpg
• https://commons.wikimedia.org/wiki/File:%D8%B4%D8%AC%D8%B1%D8%A9_%D8%B9%D8%A7%D8%A6%D9%84%D8%A9_%D8%A7%D9%84%D8%B9%D8%AC%D8%A7%D8%AC%D9%8A.jpg
• https://commons.wikimedia.org/wiki/File:Jamadom.jpg
• https://commons.wikimedia.org/wiki/File:Sophie_Le_Ray.jpg
• https://commons.wikimedia.org/wiki/File:Scott_Ragsdale.jpg
• https://commons.wikimedia.org/wiki/File:Bordercontrolslo.jpg
• https://commons.wikimedia.org/wiki/File:Simon_Quendlers_Ausstellung_%27Reaktionen%27_im_Bank_Austria_Kunstforum_Wien.jpg
• https://commons.wikimedia.org/wiki/File:Simon_Quendler_Plakat_zur_Austtellung_%27Reaktionen%27_im_Bank_Austria_Kunstforum_Wien.jpg
• https://commons.wikimedia.org/wiki/File:Acuiferodeltorrente.jpg.jpg

I have also noticed the broken files. I tagged the first one as a broken file on the 26th Feb - https://commons.wikimedia.org/wiki/File:Hamptons_in_the_Summer.jpg

PNG files have the same problem: https://commons.wikimedia.org/w/index.php?title=Special:Log&offset=&limit=500&type=delete&user=Jdx&page=&tagfilter=&hide_thanks_log=1&hide_patrol_log=1&hide_tag_log=1&withJS=MediaWiki%3AGadget-rightsfilter.js&lifilter=1&lifilterexpr=File%20is%20corrupt. If one wants some of these files to be restored, just please let me know.

I'm looking at the code that does this work, and I don't see any way the uploads could be failing mid-process and still getting published to the stash with missing chunks. I think this must have something to do with either connection issues, a new browser version doing slice() differently on file objects, or something I can't anticipate.

In case it is helpful to track down the problem, AFAICS all truncated files posted so far are cross-wiki uploads, non cross-wiki uploads are unaffected?

Two more, now deleted:
Folklorni ansambl "Rožaje" .jpg
Foto Fachada Faculdade Arquidiocesana de Curvelo.jpg

The problem is still continuing... I propose to switch cross-wiki uploads off and forbid it until the problem is solved. An other possibility is to set a 5MB limit to cross-wiki uploads, to prevent faulty uploads.

Related comment and discussion:

I see you made Category:Incomplete JPG files (5 MB interruption) and commented at phab:T190988. Using mw:Help:CirrusSearch#filesize, filesize:5120,5120 (exactly 5 MB) currently gives 6008 hits of all file types. Most have the bug. I found Jotzet has reuploaded cropped versions of many affected images. This sometimes seems problematic. For example, File:Barbie Store.jpg does not appear useful and does not reflect the image the author tried to upload. PrimeHunter (discussion) 10:55, 18 November 2018 (UTC)

@PrimeHunter: This bug requiered urgent actions: systems of blocks and warnings to prevent faulty uploads; immediate appeal to the uploaders to reupload the files, with an advice how to avoid the failure. Regrettably, most of the uploads are from one-time and occasional users who can never correct their uploads. Regrettably, many files were speedily deleted without any attempt to request a reupload. Regrettably, the bug continues many month without any appropriate measures. ŠJů 11:06, 18 November 2018 (UTC)

More examples: https://commons.wikimedia.org/wiki/File:East_Huntington_Bridge_Looking_Downstream.jpg and https://commons.wikimedia.org/wiki/File:East_Huntington_Looking_Downstream.jpg

In T190988#4755722, @SJu wrote:

The problem is still continuing... I propose to switch cross-wiki uploads off

That's certainly the most sensible thing to do. We don't need this unmaintained piece of software to cause havoc on our projects.

In T190988#4873910, @Nemo_bis wrote:

In T190988#4755722, @SJu wrote:

The problem is still continuing... I propose to switch cross-wiki uploads off

That's certainly the most sensible thing to do. We don't need this unmaintained piece of software to cause havoc on our projects.

I agree. We get lots of complaints traced to it at Commons talk:Abuse filter.

Another example: https://commons.wikimedia.org/wiki/File:Miguel_Baptista_Benedict.jpg

There seems to be no shift in the problem solution. Please help to correct, improve and internationalize a message template intended to inform uploaders of affected files.

This affects all types of files, not just JPEG files. There are now 5824 files in the subcategories of https://commons.wikimedia.org/wiki/Category:Incomplete_files_(5_MB_interruption).

You are probably not going to find the answer by looking at the code. More likely, it's a server configuration issue. 5 MB seems to be a popular limit for PHP configurations. With the cross-wiki uploads, there may be internal calls that get cut off at this limit. Please check your server logs.

In T190988#5180524, @Krinkle wrote:

Over 6,000 files have been permanently damaged, lost without recovery, and without the user's being informed about this or aware during the upload.

A hotfix seems reasonable. Once fixed we can contact the users, asking them to re-upload the file(s) if possible.

Change 510428 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/core@master] Don't allow completing a partial stash upload

https://gerrit.wikimedia.org/r/510428

Change 510428 merged by jenkins-bot:
[mediawiki/core@master] Don't allow completing a partial stash upload

https://gerrit.wikimedia.org/r/510428

Status?

The patch that we think will fix the problem is on production since (I think) May 23. I don't see anything obvious in the category https://commons.wikimedia.org/wiki/Category:Incomplete_files_(5_MB_interruption) from later than that. So perhaps it's fixed? Not sure how to verify

Search of exactly 5 MB uploads sorted by creation date descending. The most recent error is from 23 May. There were many daily errors before that so it looks like the issue is fixed.

Great. Ok to resolve the ticket?

Thanks both! (yes from my end, unless there is any needed follow-up, which should probably be in a separate task)

Caused T264333

Seems to be back again: @Cparle @matthiasmullie

T266903

Change 639469 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/core@master] Pass along ignorewarnings param to all individual chunks being uploaded

https://gerrit.wikimedia.org/r/639469

Change 639469 merged by jenkins-bot:
[mediawiki/core@master] Pass along ignorewarnings param to all individual chunks being uploaded

https://gerrit.wikimedia.org/r/639469

Change 639953 had a related patch set uploaded (by Tim Starling; owner: Matthias Mullie):
[mediawiki/core@wmf/1.36.0-wmf.16] Pass along ignorewarnings param to all individual chunks being uploaded

https://gerrit.wikimedia.org/r/639953

Change 639955 had a related patch set uploaded (by Tim Starling; owner: Matthias Mullie):
[mediawiki/core@wmf/1.36.0-wmf.14] Pass along ignorewarnings param to all individual chunks being uploaded

https://gerrit.wikimedia.org/r/639955

Change 639953 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.16] Pass along ignorewarnings param to all individual chunks being uploaded

https://gerrit.wikimedia.org/r/639953

Change 639955 merged by Tim Starling:
[mediawiki/core@wmf/1.36.0-wmf.14] Pass along ignorewarnings param to all individual chunks being uploaded

https://gerrit.wikimedia.org/r/639955

Change 639961 had a related patch set uploaded (by Reedy; owner: Matthias Mullie):
[mediawiki/core@REL1_35] Pass along ignorewarnings param to all individual chunks being uploaded

https://gerrit.wikimedia.org/r/639961

Change 639961 merged by jenkins-bot:
[mediawiki/core@REL1_35] Pass along ignorewarnings param to all individual chunks being uploaded

https://gerrit.wikimedia.org/r/639961

Another victim: https://commons.wikimedia.org/wiki/File:Jilemnick%C3%BD_h%C5%99bitov_Jind%C5%99ich_Ambro%C5%BE_n%C3%A1hrobek.jpg

Another victim, this time at 6MB: https://commons.wikimedia.org/wiki/File:Age_of_sail_.jpg