Page MenuHomePhabricator

API "async" upload treats all stashed files as empty (Unknown warning "emptyfile, stashfailed" in UploadWizard for files over 10 MB)
Closed, ResolvedPublic

Description

Unknown warning: "emptyfile, stashfailed" in UploadWizard when uploading large TIFF file.

I am trying to upload a 24M TIFF file. Normally it takes a number of minutes to upload. Today it uploaded in seconds. Strange. However, I proceed with adding the Source, etc, and then when I have finished the Title/Description etc, I click on Next and get
Unknown warning: "emptyfile, stashfailed".
The file looks fine on my computer (opens in photo tools and looks like I expect it to look). I have repeated the upload a few times and the error persists (as does the unusually fast upload). What is the problem here? I have tried changing the file name but the problem persists. [[User:Kerry Raymond|Kerry Raymond]]

https://commons.wikimedia.org/wiki/Commons:Village_pump#Commons_upload_failing

Event Timeline

matmarex created this task.Mar 17 2016, 6:41 PM
Restricted Application added a project: Multimedia. · View Herald TranscriptMar 17 2016, 6:41 PM
Restricted Application added subscribers: Steinsplitter, Aklapper. · View Herald Transcript
Ankry added a subscriber: Ankry.Mar 18 2016, 9:55 AM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptMar 18 2016, 9:55 AM
Ankry added a subscriber: Wieralee.Mar 18 2016, 9:55 AM
Ankry added a comment.Mar 18 2016, 9:59 AM

Same problem reported in OTRS Ticket#2016031710021599.

I tried this myself with different large TIFF files (downloaded from Commons and edited so as not to trigger the duplicate check) and I can't reproduce this.

Perhaps there's something weird about the specific files. @Ankry, can you request the OTRS person to send us a copy of the file? They can email it to me at matma.rex@gmail.com or upload through https://www.dropbox.com/request/Py854hq6b3Kp1jkHpEsI (you can use 'example' for firstname/lastname and 'example@example.com' for email). It's also possible to upload files to Phabricator at https://phabricator.wikimedia.org/file/upload/, but I'm not sure what the file size limit is, and I know my email and Dropbox can handle 100 MB+.

(I'll ask Kerry Raymond on Commons for a copy of their file.)

If that's impossible, I'll try to add some logging for this, but this will only give us any information the next time someone hits this issue.

Herzi.Pinki added a subscriber: Herzi.Pinki.EditedMar 18 2016, 11:34 PM

Same for me, got the same error for simple jpg. Chunked uploads are disabled in my config.
There seems to be a limit of 10 MB making the difference. The file in question was slightly above 10 MB, I scaled it down to less than 10 MB and than it worked.

matmarex added a subscriber: aaron.Mar 19 2016, 2:48 AM

@aaron dcb5ec5cbf92b9a07f0776b9c194183a13400193 was the only change to file handling I see between wmf.16 and wmf.17. Could it plausibly result in stashed files getting truncated? I'm really not familiar with that code.

FYI, this has just been reported on Freenode # wikimedia-commons a few hours ago

A screenshot of how this actually looks (courtesy of Commons user Dudley Miles):

It turns out we have some logging for these errors in UploadWizard, but it also turns out that it doesn't log anything useful (only the same error message as shown to the user). But at least I confirmed that this isn't specific to any browser (there are logged errors for every major one) and that it started after 1.27.0-wmf.17 deployment (the first logged instance ever is dated 2016-03-16 20:53:23, wmf.17 deployment to Commons finished 19:58).

At the moment, there are 1291 instances of this error logged (I think it counts every failed upload attempt, it's hard to tell how many users were affected because the data is somewhat anonymized) – for comparison, there were 17725 successful uploads with UploadWizard since this started.

There seems to be a limit of 10 MB making the difference. The file in question was slightly above 10 MB, I scaled it down to less than 10 MB and than it worked.

Hmm, UploadWizard uses the 'async' mode for files bigger than 10 MB.

Change 278547 had a related patch set uploaded (by Bartosz Dziewoński):
UploadBase: Set mFileSize, if given, even if mTempPath is unknown

https://gerrit.wikimedia.org/r/278547

Aha, I tracked this down to a bug in the upload backend, which caused it to treat all stashed files as if they were empty when using async uploading. Since UploadWizard uses async uploading for all files over 10 MB, this made it impossible to upload them. (This isn't UploadWizard-specific and would have also affected any other upload tool using this option, such as https://commons.wikimedia.org/wiki/User_talk:Rillke/bigChunkedUpload.js.)

The above patch should fix this, but it's definitely not going to be deployed during a weekend (and it still needs someone to code-review it). So I put a small hack in Commons' common.js, which should work around this in UploadWizard: https://commons.wikimedia.org/w/index.php?title=MediaWiki:Common.js&diff=prev&oldid=190720985. This might make uploading large files unreliable… but right now it's impossible. I'm hoping to get the proper fix deployed on Monday or Tuesday.

(I also filed T130484 about displaying a nicer error message for this in the future, T130437 and T130438 about some other problems visible in the screenshot above, and T130485 about making our logged information about errors possibly useful for debugging this kind of a problem.)

matmarex renamed this task from Unknown warning: "emptyfile, stashfailed" in UploadWizard when uploading large TIFF file to API "async" upload treats all stashed files as empty (Unknown warning "emptyfile, stashfailed" in UploadWizard for files over 10 MB).Mar 20 2016, 7:54 AM
matmarex triaged this task as High priority.
matmarex edited projects, added MediaWiki-Uploading, Regression; removed UploadWizard.

Change 278547 merged by jenkins-bot:
UploadBase: Set mFileSize, if given, even if mTempPath is unknown

https://gerrit.wikimedia.org/r/278547

Change 278724 had a related patch set uploaded (by Bartosz Dziewoński):
UploadBase: Set mFileSize, if given, even if mTempPath is unknown

https://gerrit.wikimedia.org/r/278724

Change 278724 merged by jenkins-bot:
UploadBase: Set mFileSize, if given, even if mTempPath is unknown

https://gerrit.wikimedia.org/r/278724

matmarex closed this task as Resolved.Mar 21 2016, 3:22 PM

Patch deployed. I reverted the on-wiki workaround.