Page MenuHomePhabricator
Create Task
Maniphest T355610

Problem uploading 4GB FLAC file in Upload Wizard to Wikimedia Commons
Closed, DuplicatePublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

  • Upload a Flac file of 3,93 GB using Uplad Wizard or chucked uplad

What happens?:
Show the follow error (see screenshoot in attachement):

ZOOM0002_LR-0001.flac
The file mwstore://local-swift-codfw/local-temp/3/34/1anr0q9u0hgg.6o1jm5.32521.flac.0 does not exist.

What should have happened instead?:

The file should be uploaded to commons

Software version (skip for WMF-hosted wikis like Wikipedia):
Wikimedia commons

Other information (browser name/version, screenshots, etc.):
Tested on Brave and Chrome browsers
I ruled out connection problems, I tried to do this several times with several computers, I also ruled out any problems with the file

See also: T355433: Problem with uploading large files (2 GB)

Related Objects

Event Timeline

Wilfredor created this task.Jan 22 2024, 9:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 22 2024, 9:59 PM
Wilfredor added a comment.Jan 22 2024, 10:00 PM

Screenshot 2024-01-22 at 4.55.55 PM.png (1,558×634 px, 90 KB)

Bugreporter added a project: SRE-swift-storage.Jan 23 2024, 12:52 AM
Bugreporter updated the task description. (Show Details)Jan 23 2024, 7:30 AM
Bugreporter mentioned this in T355433: Problem with uploading large files (2 GB).
MatthewVernon subscribed.Jan 23 2024, 10:25 AM

I don't think this is a result of a swift failure, so we'd need input from the upload wizard folks. Looking in the swift logs, I see:

moss-fe2001.codfw.wmnet: Jan 22 20:05:44 moss-fe2001 proxy-server: 10.192.16.173 10.192.32.51 22/Jan/2024/20/05/44 PUT /v1/AUTH_mw/wikipedia-commons-local-temp.34/3/34/1anr0q9u0hgg.6o1jm5.32521.flac.0 HTTP/1.0 201 - wikimedia/multi-ht\
tp-client%20v1.1 AUTH_tkda8d41a28... - - - tx4d6add5273da4fe991a4c-0065aeca98 - 0.2564 - - 1705953944.711836100 1705953944.968250513 -
ms-fe1011.eqiad.wmnet: Jan 22 20:05:45 ms-fe1011 proxy-server: 10.192.16.173 10.64.32.182 22/Jan/2024/20/05/45 PUT /v1/AUTH_mw/wikipedia-commons-local-temp.34/3/34/1anr0q9u0hgg.6o1jm5.32521.flac.0 HTTP/1.0 201 - wikimedia/multi-http-c\
lient%20v1.1 AUTH_tked0f10995... - - - txb32af9337d4d439980e16-0065aeca98 - 0.3465 - - 1705953944.986651659 1705953945.333140373 -
#
moss-fe2001.codfw.wmnet: Jan 22 21:12:39 moss-fe2001 proxy-server: 10.194.185.231 10.192.32.51 22/Jan/2024/21/12/39 GET /v1/AUTH_mw/wikipedia-commons-local-temp.34/3/34/1anr0q9u0hgg.6o1jm5.32521.flac.0 HTTP/1.0 200 - wikimedia/multi-h\
ttp-client%20v1.1 AUTH_tkda8d41a28... - 5242880 - tx4f4d2f80f39f40e995b35-0065aeda47 - 0.0680 - - 1705957959.490641356 1705957959.558657646 0
#
moss-fe2001.codfw.wmnet: Jan 22 21:13:31 moss-fe2001 proxy-server: 10.194.185.231 10.192.32.51 22/Jan/2024/21/13/31 DELETE /v1/AUTH_mw/wikipedia-commons-local-temp.34/3/34/1anr0q9u0hgg.6o1jm5.32521.flac.0 HTTP/1.0 204 - wikimedia/mult\
i-http-client%20v1.1 AUTH_tkda8d41a28... - - - tx2a67a20aa92f45a7a9bdc-0065aeda7b - 0.1023 - - 1705958011.038794994 1705958011.141051054 0
ms-fe1010.eqiad.wmnet: Jan 22 21:13:39 ms-fe1010 proxy-server: 10.194.185.231 10.64.16.92 22/Jan/2024/21/13/39 DELETE /v1/AUTH_mw/wikipedia-commons-local-temp.34/3/34/1anr0q9u0hgg.6o1jm5.32521.flac.0 HTTP/1.0 204 - wikimedia/multi-htt\
p-client%20v1.1 AUTH_tked0f10995... - - - tx84343ef96c014837b9a91-0065aeda83 - 0.0306 - - 1705958019.188747406 1705958019.219331980 0
#
ms-fe2013.codfw.wmnet: Jan 22 21:16:39 ms-fe2013 proxy-server: 10.194.156.0 10.192.0.87 22/Jan/2024/21/16/39 GET /v1/AUTH_mw/wikipedia-commons-local-temp.34/3/34/1anr0q9u0hgg.6o1jm5.32521.flac.0 HTTP/1.0 404 - wikimedia/multi-http-cli\
ent%20v1.1 AUTH_tkda8d41a28... - 70 - txfc24b7cafbed42c1932a3-0065aedb37 - 0.0280 - - 1705958199.731104851 1705958199.759110689 0
ms-fe2013.codfw.wmnet: Jan 22 21:16:43 ms-fe2013 proxy-server: 10.194.156.0 10.192.0.87 22/Jan/2024/21/16/43 GET /v1/AUTH_mw/wikipedia-commons-local-temp.34/3/34/1anr0q9u0hgg.6o1jm5.32521.flac.0 HTTP/1.0 404 - wikimedia/multi-http-cli\
ent%20v1.1 AUTH_tkda8d41a28... - 70 - txfe682317f4f84fda85075-0065aedb3b - 0.0282 - - 1705958203.732530355 1705958203.760778427 0

So you can see the file is uploaded to both swift clusters from mw2320 at 20:05; there is a successful GET at 21:12 followed by a successful DELETE from both clusters at 21:13 (from the same client IP as did the GET). Then at 21:16 there are two GETs that return 404 (which is what you'd expect after a DELETE); I presume the UI message you get above is as a result of one of those 404s.

Wilfredor added a comment.Jan 23 2024, 10:41 AM

Apparently it is not a storage problem but in the way the files are unified in a subsequent process, the temporary files are deleted without first confirming that the final destination file is created satisfactorily.

MatthewVernon mentioned this in T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..Feb 21 2024, 11:31 AM
Bawolff subscribed.Feb 22 2024, 12:03 AM

https://gerrit.wikimedia.org/r/1005632 may help with this.

Bawolff closed this task as a duplicate of T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..Feb 22 2024, 12:12 AM
Aklapper renamed this task from Problem uploading FLAC file in Upload Wizzard to Wikimedia Commons to Problem uploading 4GB FLAC file in Upload Wizard to Wikimedia Commons.Mar 17 2024, 7:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits