Page MenuHomePhabricator

FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0".
Open, NormalPublic

Description

Trying to upload T200223: Server side upload for -revi, I miserably failed (at least twice IIRC) with the warning:

FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0".

I have no idea what's wrong with my file here <,< And I think this is partially reason that I can't see UploadStash page? (It 503s)

Event Timeline

revi created this task.Jul 31 2018, 4:54 PM
Restricted Application added a project: Multimedia. · View Herald TranscriptJul 31 2018, 4:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
revi moved this task from Incoming to Radar on the User-revi board.Jul 31 2018, 4:54 PM
revi updated the task description. (Show Details)Aug 1 2018, 5:28 AM
revi added a subscriber: Bawolff.

Odd UploadStash would fail, can you try again? Does the 503 report any further error?

I tried once again, actually stash was empty at the moment (meh) so I re-tried the upload with the expectation that the upload will fail (and UploadStash will fail again), but it managed to upload it successfully this time (meh).

Ack, thanks for the report @revi ! I'll defer to some people more intimate with uploadstash

Yann added a subscriber: Yann.EditedDec 16 2018, 11:02 AM

I got the same error while trying to upload a file with Rillke's Chunked Upload script on https://commons.wikimedia.org/wiki/File:L%27Illustration,_mars-ao%C3%BBt_1849.pdf
( https://commons.wikimedia.org/wiki/User:Rillke/bigChunkedUpload.js )

03592: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/2/2a/16981zp2ivt0.wq8wbr.1.pdf.0".

I think the file is in Upload stash: 16986mjwpyvc.x9ltso.1.pdf

And it is the second time, it happened today. There is another file stucked in Upload stash: 1697wll1czuk.rcpwyj.1.pdf

Tgr added a subscriber: Tgr.EditedDec 16 2018, 8:37 PM

That error message is backend-fail-read in FileBackendStore::doConcatenate() I believe. The earliest error in the request is Memcached error for key "/*/mw-wan/WANCache:v:filebackend:local-swift-eqiad:wikipedia-commons:file:<hash>" on server "127.0.0.1:11213": A TIMEOUT OCCURRED; that key seems to come from FileBackendStore::fileCacheKey(). Tracing back code paths in FileBackend is not really feasible (we should probably add a backtrace to the memcached error) but I'm guessig something in FileBackendStore or SwiftFileBackend messes up cache error handling and interprets "file stat not in cache" as "file does not exist".

Yann added a comment.Jan 4 2019, 12:22 PM

I could upload the 3 files which caused this bug.

Koavf added a subscriber: Koavf.Jan 28 2019, 8:26 PM

Just received the same error on Commons for File:116th United States Congress House Floor - 2019-01-10.webm. Trying again.

Koavf added a comment.Jan 28 2019, 9:09 PM

It failed twice. :/ I guess I am going to use video2commons and then make a Phabricator tag if (when) it fails again. </3