Uploads timing out with PWB and upload wizard
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T292954 [epic] large file uploads to commons | |||
| Open | None | T278104 Unable to upload to Commons: uploadstash-file-not-found: Key "187kyl5ozj74.xtav8j.51508.djvu" not found in stash | |||
| Resolved | Legoktm | T294689 Please move file from en.wp to commons |
Event Timeline
Comment Actions
@AlexisJazz Pywikibot. This is to do with some of my own scripts (which is why it's only in my own Project). I have been unable to upload large files for over a week. Verbose Pywikibot indicates this is to do with "stash":
pywikibot.data.api.APIError: uploadstash-file-not-found: Key "187kyl5ozj74.xtav8j.51508.djvu" not found in stash
I am unsure if this is a problem with anything other than my own scripts or files.
Comment Actions
What happens when you try to upload a large file with UploadWizard?
https://commons.wikimedia.org/wiki/Commons:Village_pump/Technical#Stash_has_not_uploaded_for_days and https://commons.wikimedia.org/wiki/Commons:Administrators%27_noticeboard#Stash_has_not_uploaded_for_days by @Koavf is probably related.
Comment Actions
@AlexisJazz It uploads incredibly slowly (like 50kBps), just like Pywikibot and eventually fails with simply "error".
BTW, I have tried Pywikibot with no chunking (413 payload too large), 1MB, 10MB and 100MB chunks.
Comment Actions
Betacommons seems to fail as well. Tried to upload a 100MB+ file with UploadWizard, after the progress bar was full it said "processing file" and now just says "The server did not respond within the expected time." and "None of the uploads were successful."
Comment Actions
I've successfully uploaded several <100MB files in the time period. The one >100MB file I've tried fails (I've been blindly trying different things so exact failure symptoms are a bit vague). All uploads with bigChunkedUpload.js with stash/async deselected.
The failing file is here. It seems to have somehow been published from stash with only the first ~20MB chunk included (twice!), but the upload as such completed all ~30 chunks and only failed at the end of the final chunk (but was obviously still visible in the stash since I was able to publish it).
I've also felt all uploads are slower than expected, but I've not tried to quantify it (too many variables).
Comment Actions
Ok, testing the >100MB file locally on enWS (I think most of the relevant bits of the stack are the same as for Commons), bigChunkedUpload.js tells me "Upload is stuck" for every single chunk (32 x 20MB chunks) but then seems to recover. After the last chunk hits 100% it tells me "Server error 0 after uploading chunk:" (I think this is an empty response from the server). After waiting and retrying a couple more times it terminates with the message "FAILED: internal_api_error_DBQueryError: [91f56af6-cec2-4969-938f-3aeaf9f35aff] Caught exception of type Wikimedia\Rdbms\DBQueryError" which I'm pretty certain is coming from somewhere inside MW proper rather than from Rillke's code.
An hour later the file is still sitting in my upload stash.
Comment Actions
Just to note: the file I tried to upload to betacommons was a video file. (so no PDF or DJVU processing)
Xover, iirc 100MB is the limit for "basic" uploads (like Special:Upload) without using the stashing/reconstructing stuff like UploadWizard and bigChunkedUpload.js can.
Comment Actions
@Aklapper I'm not entirely steady on the projects/components and their scope, so apologies if I'm hopelessly confused, but looking at the descriptions for them I would say this task falls under MediaWiki-Uploading and UploadWizard? Or is this obviously pinpointed somewhere down in the Swift part of the stack? And maybe UploadWizard is excluded since this happens via API upload too?
Also, I see T278389 has vaguely similar symptoms and may conceivably be related (even if the error message is pretty generic).
Comment Actions
I've been getting a similar error with Pattypan all day.
Error 1) Error: Key not found in stash
Error 2) Error: Could not read file mwstore://local-swift-equiad-temp/c/c5/...
Error 3) Error: Server returned HTTP response code: 500 for URL: https:/commons.wikimedia.org/w/api.php?format=xml&maxlag=5&action=upload
I've been trying to upload ~120mb tif.
Comment Actions
Copy of my comment on T292954#7469361:
Using bigChunkedUpload.js I uploaded a 921MB video:
01006: 224/231> Chunk uploaded 01006: 225/231> in progress Upload: 99% 01011: 225/231> Chunk uploaded 01011: 226/231> in progress Upload: 100% 01016: 226/231> Chunk uploaded 01016: 227/231> in progress Upload: 99% 01020: 227/231> Chunk uploaded 01020: 228/231> in progress Upload: 100% 01024: 228/231> Chunk uploaded 01025: 229/231> in progress Upload: 100% 01029: 229/231> Chunk uploaded 01029: 230/231> in progress Upload: 98% 01033: 230/231> Chunk uploaded 01033: 231/231> in progress Upload: 93% 01043: 231/231> upload is stuck 01044: 231/231> Connection seems to be okay. Waiting one more time... 01048: 231/231> upload is stuck 01049: 231/231> Connection seems to be okay. Waiting one more time... 01053: 231/231> upload is stuck 01054: 231/231> Connection seems to be okay. Waiting one more time... 01058: 231/231> upload is stuck 01059: 231/231> Connection seems to be okay. Waiting one more time... 01063: 231/231> upload is stuck 01064: 231/231> Connection seems to be okay. Waiting one more time... 01068: 231/231> upload is stuck 01069: 231/231> Connection seems to be okay. Waiting one more time... 01073: 231/231> upload is stuck 01074: 231/231> Connection seems to be okay. Waiting one more time... 01078: 231/231> upload is stuck 01079: 231/231> Connection seems to be okay. Waiting one more time... 01083: 231/231> upload is stuck 01084: 231/231> Connection seems to be okay. Waiting one more time... 01088: 231/231> upload is stuck 01089: 231/231> Connection seems to be okay. Waiting one more time... 01093: 231/231> upload is stuck 01094: 231/231> Server error 0 after uploading chunk: Response: 01094: 231/231> Connection seems to be okay. Re-sending this request. 01094: 231/231> Connection seems to be okay. Re-sending this request. Upload: 96% 01095: FAILED: uploadstash-file-not-found: Key "18qfe2nwiafg.dx1sfn.2521907.webm" not found in stash.
Special:UploadStash showed:
18qfff9yoto0.mcayum.2521907.webm (view thumbnail)
I entered this:
var a = new mw.Api(); a.postWithToken( 'edit', { action: 'upload', filekey: '18qfff9yoto0.mcayum.2521907.webm', 'filename': 'David Pakman - Politics, media and reality tunnels.webm' } );And got an empty response. The file was successfully published to File:David Pakman - Politics, media and reality tunnels.webm. I've since moved it to File:David Pakman talks to Rebel Wisdom about politics and media.webm because I discovered the similarly named File:David Pakman - Politics, Media and Reality Tunnels.webm already existed.
Why the UploadStash key changed? Beats me
Comment Actions
HI, I got a similar error while trying to import a 492 MB TIFF via chunked-upload from https://archive.org/details/clevelandart-1958.20-the-brook :
Direct link https://archive.org/download/clevelandart-1958.20-the-brook/1958.20_full.tif
FAILED: uploadstash-file-not-found: Key "18qxmndl3k3w.tobuo5.1.tif" not found in stash.
but https://commons.wikimedia.org/wiki/Special:UploadStash shows
18qxmndl3k3w.tobuo5.1.tif
Then I tried on Chrome console
var a = new mw.Api(); a.postWithToken( 'edit', { action: 'upload', filekey: '18qxmndl3k3w.tobuo5.1.tif', 'filename': 'Eau_courante_en_sous-bois,_par_Paul_Cézanne,_Cleveland_Museum_of_Art.tiff' } );and I got
Uncaught ReferenceError: mw is not defined
at <anonymous>:1:9Fetching thumbnail failed: Array ( [0] => Array ( [0] => http-bad-status [1] => 404 [2] => Not Found ) ) URL = http://thumbor.svc.eqiad.wmnet:8800/wikipedia/commons/thumb/temp/f/f7/20211104221337%21chunkedupload_cb2ef83b66fa.tif/-20211104221337%21chunkedupload_cb2ef83b66fa.tif.png