Page MenuHomePhabricator

Unable to upload to Commons: uploadstash-file-not-found: Key "187kyl5ozj74.xtav8j.51508.djvu" not found in stash
Open, Needs TriagePublic

Description

Uploads timing out with PWB and upload wizard

Event Timeline

Inductiveload triaged this task as Unbreak Now! priority.Mar 22 2021, 10:39 AM
AlexisJazz lowered the priority of this task from Unbreak Now! to Needs Triage.Mar 22 2021, 12:08 PM
AlexisJazz added a subscriber: AlexisJazz.

What's "PWB"?

@AlexisJazz Pywikibot. This is to do with some of my own scripts (which is why it's only in my own Project). I have been unable to upload large files for over a week. Verbose Pywikibot indicates this is to do with "stash":

pywikibot.data.api.APIError: uploadstash-file-not-found: Key "187kyl5ozj74.xtav8j.51508.djvu" not found in stash

I am unsure if this is a problem with anything other than my own scripts or files.

@AlexisJazz It uploads incredibly slowly (like 50kBps), just like Pywikibot and eventually fails with simply "error".

BTW, I have tried Pywikibot with no chunking (413 payload too large), 1MB, 10MB and 100MB chunks.

Betacommons seems to fail as well. Tried to upload a 100MB+ file with UploadWizard, after the progress bar was full it said "processing file" and now just says "The server did not respond within the expected time." and "None of the uploads were successful."

I've successfully uploaded several <100MB files in the time period. The one >100MB file I've tried fails (I've been blindly trying different things so exact failure symptoms are a bit vague). All uploads with bigChunkedUpload.js with stash/async deselected.

The failing file is here. It seems to have somehow been published from stash with only the first ~20MB chunk included (twice!), but the upload as such completed all ~30 chunks and only failed at the end of the final chunk (but was obviously still visible in the stash since I was able to publish it).

I've also felt all uploads are slower than expected, but I've not tried to quantify it (too many variables).

Ok, testing the >100MB file locally on enWS (I think most of the relevant bits of the stack are the same as for Commons), bigChunkedUpload.js tells me "Upload is stuck" for every single chunk (32 x 20MB chunks) but then seems to recover. After the last chunk hits 100% it tells me "Server error 0 after uploading chunk:" (I think this is an empty response from the server). After waiting and retrying a couple more times it terminates with the message "FAILED: internal_api_error_DBQueryError: [91f56af6-cec2-4969-938f-3aeaf9f35aff] Caught exception of type Wikimedia\Rdbms\DBQueryError" which I'm pretty certain is coming from somewhere inside MW proper rather than from Rillke's code.

An hour later the file is still sitting in my upload stash.

AlexisJazz renamed this task from Investigate broken upload to Commons to Investigate broken upload to Commons: FAILED: internal_api_error_DBQueryError: Caught exception of type Wikimedia\Rdbms\DBQueryError.Mar 22 2021, 6:36 PM

Just to note: the file I tried to upload to betacommons was a video file. (so no PDF or DJVU processing)

Xover, iirc 100MB is the limit for "basic" uploads (like Special:Upload) without using the stashing/reconstructing stuff like UploadWizard and bigChunkedUpload.js can.

Aklapper renamed this task from Investigate broken upload to Commons: FAILED: internal_api_error_DBQueryError: Caught exception of type Wikimedia\Rdbms\DBQueryError to Unable to upload to Commons: uploadstash-file-not-found: Key "187kyl5ozj74.xtav8j.51508.djvu" not found in stash.Mar 28 2021, 5:08 PM

@Aklapper I'm not entirely steady on the projects/components and their scope, so apologies if I'm hopelessly confused, but looking at the descriptions for them I would say this task falls under MediaWiki-Uploading and UploadWizard? Or is this obviously pinpointed somewhere down in the Swift part of the stack? And maybe UploadWizard is excluded since this happens via API upload too?

Also, I see T278389 has vaguely similar symptoms and may conceivably be related (even if the error message is pretty generic).

I've been getting a similar error with Pattypan all day.

Error 1) Error: Key not found in stash
Error 2) Error: Could not read file mwstore://local-swift-equiad-temp/c/c5/...
Error 3) Error: Server returned HTTP response code: 500 for URL: https:/commons.wikimedia.org/w/api.php?format=xml&maxlag=5&action=upload

I've been trying to upload ~120mb tif.