Page MenuHomePhabricator

Uploading large files to Commons almost always fails
Open, Needs TriagePublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

What happens?:
The upload seldom succeeds. The server always returns 503 or 504.

What should have happened instead?:
The upload succeeds.

Software version (skip for WMF-hosted wikis like Wikipedia):
Wikimedia Commons
mwclient / 0.10.1

Other information (browser name/version, screenshots, etc.):
The problem exists for years. Dating back to some weeks in 2021, I cannot even upload files of > 200MiB files.

Until now, the failure rate of uploading files is proportional to the size of the files being uploaded. That is, uploading a ~ 1GiB file fails sometimes, ~ a 2GiB file fails frequently and ~ a 3GiB file fails almost all the time.

The failure appears to have nothing to do with the network. Uploading from my local box, a third-party Cloud server, Toolforge, or WMCS Cloud VPS yields the same result in general.

Sometimes, I manage to upload failed files manually via Special:UploadWizard. It is slightly more stable, but still suffers from the issue.

Logs (some repeated lines omitted):

INFO:__main__:Uploading File:YNUTCM-294300021-5御纂醫宗金鑒(卷四)72494.pdf (2325273937 B)                                                      
INFO:__main__:Retrying <function main.<locals>.do1 at 0x7fdb56345580>                                                                   
WARNING:mwclient.client:Received 503 response: <!DOCTYPE html>                                                                                    
<html lang="en">                                                                                                                                  
<meta charset="utf-8">                                                                                                                            
<title>Wikimedia Error</title>                                                                                                                    
...<h1>Error</h1>                                                                                                                                    
<div class="content-text">                                                                                                                        
<p>Our servers are currently under maintenance or experiencing a technical problem.                                                               
...<div class="footer"><p>If you report this error to the Wikimedia System Administrators, please include the details below.</p><p class="text-muted"
><code>Request from 172.16.4.85 via cp1079 cp1079, Varnish XID 720477065<br>Error: 503, Backend fetch failed at Sat, 01 Jul 2023 04:59:00 GMT</cod
e></p>                                                                                                                                            
</div>                                                                                                                                            
</html>                                                                                                                                           
. Retrying in a moment.                                                                                                                           
INFO:__main__:Retrying <function main.<locals>.do1 at 0x7fdb56345580>  
...
  File "/root/.cache/pypoetry/virtualenvs/uploader-URSXssvu-py3.11/lib/python3.11/site-packages/mwclient/client.py", line 679, in upload
    return self.chunk_upload(file, filename, ignore, comment, text)                                                                               
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                               
  File "/root/.cache/pypoetry/virtualenvs/uploader-URSXssvu-py3.11/lib/python3.11/site-packages/mwclient/client.py", line 760, in chunk_upload
    if self.handle_api_result(info, kwargs=params, sleeper=sleeper):                                                                              
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                               
  File "/root/.cache/pypoetry/virtualenvs/uploader-URSXssvu-py3.11/lib/python3.11/site-packages/mwclient/client.py", line 331, in handle_api_resul
t
    raise errors.APIError(info['error']['code'],
mwclient.errors.APIError: ('chunk-too-small', 'Minimum chunk size is 1,024 bytes for non-final chunks.', 'See https://commons.wikimedia.org/w/api.
php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.
lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes.')
...

Event Timeline

Hoi renamed this task from Upload large files to Commons almost always failed to Uploading large files to Commons almost always failed .Jul 1 2023, 6:38 AM
Reedy renamed this task from Uploading large files to Commons almost always failed to Uploading large files to Commons almost always fails.Jul 1 2023, 9:05 AM

See T340917. I can't upload files bigger than 350 MB on the English Wikisource. Systematic fail.

Sometimes, after failed uploads, I got [...] 202?-??-?? ??:??:??: Fatal exception of type "Wikimedia\RequestTimeout\RequestTimeoutException when accessing [[c:Special:UploadStash]]. I suppose it might be related.

As suggested by @Midleading, I switched to pywikibot from mwclient. The stability when uploading files of GiB magnitude has improved substantially, under async mode.