Page MenuHomePhabricator

"Could not read file" error during upload
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

This issue has been happening consistently when I attempt to upload a file to enwiki using the Rillke script. The file is a 4.71 GiB video.

What happens?:

Every chunk appears to upload successfully. The API reports that the upload job is queued, then that it is assembling. However, the job status gets errored out quickly. The API gives an HTTP 200 response with a stashfailed error and a message "Could not read file "mwstore://local-swift-eqiad/local-temp/[file path]"".

What should have happened instead?:

No error, and the upload should be successfully assembled.

Other information (browser name/version, screenshots, etc.):

Sample API responses:

{"error":{"code":"stashfailed","info":"Could not read file \"mwstore://local-swift-eqiad/local-temp/b/b6/1cp9qih7a3qg.wgwy8u.20444050.webm.24\".","*":"See https://en.wikipedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes."},"servedby":"mw-api-ext.eqiad.main-799bf4c565-gkjx9"}

{"error":{"code":"stashfailed","info":"Could not read file \"mwstore://local-swift-eqiad/local-temp/9/99/1cpckth1xpqo.n9iup.20444050.webm.34\".","*":"See https://en.wikipedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes."},"servedby":"mw-api-ext.eqiad.main-7675448b98-fnzjw"}

Event Timeline

I've gone looking in swift logs for the first of these two objects, and I find these four hits, in time order:

Jun  5 00:53:39 ms-fe1017 proxy-server: 10.67.223.8 10.64.0.188 05/Jun/2026/00/53/39 PUT /v1/AUTH_mw/wikipedia-en-local-temp.b6/b/b6/1cp9qih7a3qg.wgwy8u.20444050.webm.24 HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tkd31429ce8... 19439616 - dde7c619af0ac235d88742fb6eb489a7 tx1f2b5b340ed845af96311-006a221e13-f683416a-f311-41db-8397-31ce7b65 - 0.2584 - - 1780620819.034368515 1780620819.292721748 0
Jun  5 01:06:22 ms-fe1020 proxy-server: 10.67.207.169 10.64.179.7 05/Jun/2026/01/06/22 GET /v1/AUTH_mw/wikipedia-en-local-temp.b6/b/b6/1cp9qih7a3qg.wgwy8u.20444050.webm.24 HTTP/1.0 499 - wikimedia/multi-http-client%20v1.1 AUTH_tkd31429ce8... - 65536 - tx1caf58a560354aa499aa6-006a222108-44dccfa6-ba05-4e38-a9c6-c187f1a7 - 5.7017 - - 1780621576.440272093 1780621582.141992092 0
Jun  5 01:07:19 ms-fe1013 proxy-server: 10.67.220.5 10.64.179.6 05/Jun/2026/01/07/19 GET /v1/AUTH_mw/wikipedia-en-local-temp.b6/b/b6/1cp9qih7a3qg.wgwy8u.20444050.webm.24 HTTP/1.0 200 - wikimedia/multi-http-client%20v1.1 AUTH_tkd31429ce8... - 19439616 - txfc6c57deedc84f57af311-006a222146-44dccfa6-ba05-4e38-a9c6-c187f1a7 - 1.6357 - - 1780621638.248369932 1780621639.884046316 0
Jun  5 01:12:27 ms-fe1015 proxy-server: 10.67.220.5 10.64.158.3 05/Jun/2026/01/12/27 DELETE /v1/AUTH_mw/wikipedia-en-local-temp.b6/b/b6/1cp9qih7a3qg.wgwy8u.20444050.webm.24 HTTP/1.0 204 - wikimedia/multi-http-client%20v1.1 AUTH_tkd31429ce8... - - - tx8de76737374747b583f0a-006a22227b-44dccfa6-ba05-4e38-a9c6-c187f1a7 - 0.0813 - - 1780621947.821367741 1780621947.902695179 0

So there's a successful upload at 00:53:39, a GET request which the client gives up on (499 is client timeout) followed by a successful GET a minute later. Then, five minutes after that, the object is deleted.

I don't know what the 'Rillke script' is or does, but I don't think I can help any further with this request. AFAICT the client was able to read the uploaded object successfully, and then deleted it.

The script I'm referring to is the one at commons:Help:Chunked upload. To my knowledge, it's the only tool used for chunked uploads outside of Commons. In the past it has always handled retrying chunks that time out, and the last change was to a dependency way back in 2021.

I'll re-attempt the upload to verify where the delete request is coming from and whether there's a reupload happening after whichever chunk causes the error.

I retried the upload and it worked this time.

The previous failures happened at least like four different times over three different days, so I have no idea what could've happened differently during this attempt. I'll take it though. The file in question is here on enwiki in case that helps replicate, but unless this error starts showing up again with another file, I'm not planning on trying to reproduce the issue. As you can see from the logs, each attempt takes a while.

(Apologies Matt on the earlier comment, I went to double-check those were private IPs and completely misread the result.)

MatthewVernon claimed this task.

I'm glad to hear it worked this time! I'll close this ticket, then. If it starts happening again, do feel free to open a fresh ticket (rough timestamp and the name of the failing object like you provided this time lets me at least look at the swift logs).

[And thanks for the apology, I appreciate it - I got your comment by email with the complaint about IP addresses and had a mild panic, even though I thought I'd checked the IPs were all private before posting the log snippets. Phew :) ]