Page MenuHomePhabricator
Create Task
Maniphest T288481

Unexpected upload speed to commons
Closed, ResolvedPublic
Actions

Description

I uploaded a ~1GB video to commons today. It was very slow.

It seems the connection reaches a ~1.5 MiB/second speed and then drops to almost 0 KiB/second, then loops the pattern again and again.
Here is a diagram showing my system bandwidth during the upload:

image.png (495×932 px, 54 KB)

I tried capturing what my browser is doing (ff 78.12.0esr if that matters) during the upload:

image.png (998×1 px, 396 KB)

It took more than 30 minutes to upload the file. I don't think that's the expected behavior.

For comparison, I can upload the same file to other platforms (rsync,net, youtube, dropbox, google photos, etc) at steady rate of ~10MiB/s. And some services even faster, I guess because they have upload facilities in EU.

Moreover, the upload eventually failed entirely:

image.png (566×1 px, 83 KB)

Related Objects
Search...

StatusSubtypeAssignedTask
 InvalidNoneT292954 [epic] large file uploads to commons
 ResolvedaborreroT288481 Unexpected upload speed to commons

Event Timeline

aborrero created this task.Aug 9 2021, 8:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 9 2021, 8:03 PM
RhinosF1 subscribed.Aug 9 2021, 8:17 PM
aborrero updated the task description. (Show Details)Aug 9 2021, 8:58 PM
aborrero added a comment.Aug 10 2021, 2:09 PM

I also tried using the pywikibot upload script, with similar result.

This time, however, the script mentions action 'upload', server said: ('internal_api_error_DBQueryError', '[d8e17fc9-2337-47e5-b482-e48873888722] Caught exception of type Wikimedia\\Rdbms\\DBQueryError'

[08/10/21 13:02:11] INFO     [<Anonymous> @ commons.wikimedia.org]: Attempting login for Arturoborrero                                                                                                  wiki.py:308
[08/10/21 13:02:13] INFO     [Arturoborrero @ commons.wikimedia.org]: Fetching user rights for 'Arturoborrero'                                                                                          wiki.py:575
                    INFO     [Arturoborrero @ commons.wikimedia.org]: Successfully logged in as 'Arturoborrero'                                                                                         wiki.py:311
                    INFO     [Arturoborrero @ commons.wikimedia.org]: Fetching a list of acceptable file upload extensions.                                                                             wiki.py:717
[08/10/21 13:02:14] INFO     [Arturoborrero @ commons.wikimedia.org]: Uploading 'Dactylopterus Volitans/1.webm' to 'Dactylopterus Volitans 1 2021-08-10.webm'                                           wiki.py:356
[08/10/21 13:09:58] INFO     [Arturoborrero @ commons.wikimedia.org]: Uploading chunk 114 of 289 from 'Dactylopterus Volitans/1.webm'                                                                waction.py:220
[08/10/21 13:10:03] INFO     [Arturoborrero @ commons.wikimedia.org]: Uploading chunk 115 of 289 from 'Dactylopterus Volitans/1.webm'                                                                waction.py:220
[08/10/21 13:10:07] INFO     [Arturoborrero @ commons.wikimedia.org]: Uploading chunk 116 of 289 from 'Dactylopterus Volitans/1.webm'                                                                waction.py:220
[..]
[..]
                    ERROR    [Arturoborrero @ commons.wikimedia.org]: No response from     waction.py:43
                             server while trying to perform action 'upload'                             
                    WARNING  [Arturoborrero @ commons.wikimedia.org]: Encountered error   waction.py:229
                             while uploading, this was 1/5                                              
                    INFO     [Arturoborrero @ commons.wikimedia.org]: Unstashing          waction.py:184
                             '18jktcywt4s4.d0irq5.4547358.webm' as 'Dactylopterus                       
                             Volitans 1 2021-08-10.webm'                                                
[08/10/21 13:23:28] ERROR    [Arturoborrero @ commons.wikimedia.org]: Failed to perform    waction.py:48
                             action 'upload', server said:                                              
                             ('internal_api_error_DBQueryError',                                        
                             '[550ef302-654b-4fd1-8245-2cc27fe20768] Caught exception of                
                             type Wikimedia\\Rdbms\\DBQueryError')                                      
                    WARNING  [Arturoborrero @ commons.wikimedia.org]: Unstash failed,     waction.py:189
                             this is a attempt 1 of 5. Sleeping 5s...                                   
[08/10/21 13:23:53] ERROR    [Arturoborrero @ commons.wikimedia.org]: Failed to perform    waction.py:48
                             action 'upload', server said:                                              
                             ('internal_api_error_DBQueryError',                                        
                             '[232edab7-9025-4527-ac50-65cab7dee83f] Caught exception of                
                             type Wikimedia\\Rdbms\\DBQueryError')                                      
                    WARNING  [Arturoborrero @ commons.wikimedia.org]: Unstash failed,     waction.py:189
                             this is a attempt 2 of 5. Sleeping 5s...                                   
[08/10/21 13:24:19] ERROR    [Arturoborrero @ commons.wikimedia.org]: Failed to perform    waction.py:48
                             action 'upload', server said:                                              
                             ('internal_api_error_DBQueryError',                                        
                             '[5cbc5789-46bd-47b6-8b36-3d76c34371f7] Caught exception of                
                             type Wikimedia\\Rdbms\\DBQueryError')                                      
                    WARNING  [Arturoborrero @ commons.wikimedia.org]: Unstash failed,     waction.py:189
                             this is a attempt 3 of 5. Sleeping 5s...                                   
[08/10/21 13:24:44] ERROR    [Arturoborrero @ commons.wikimedia.org]: Failed to perform    waction.py:48
                             action 'upload', server said:                                              
                             ('internal_api_error_DBQueryError',                                        
                             '[3648d8c7-4e4e-4b4c-870e-eca557ade439] Caught exception of                
                             type Wikimedia\\Rdbms\\DBQueryError')                                      
                    WARNING  [Arturoborrero @ commons.wikimedia.org]: Unstash failed,     waction.py:189
                             this is a attempt 4 of 5. Sleeping 5s...                                   
[08/10/21 13:25:09] ERROR    [Arturoborrero @ commons.wikimedia.org]: Failed to perform    waction.py:48
                             action 'upload', server said:                                              
                             ('internal_api_error_DBQueryError',                                        
                             '[d8e17fc9-2337-47e5-b482-e48873888722] Caught exception of                
                             type Wikimedia\\Rdbms\\DBQueryError')                                      
                    WARNING  [Arturoborrero @ commons.wikimedia.org]: Unstash failed,     waction.py:189
                             this is a attempt 5 of 5. Sleeping 5s...                                   
[08/10/21 13:25:14] WARNING  Failed to upload 1 files: [PosixPath('Dactylopterus         __main__.py:109
                             Volitans/1.webm')]                                                         
[08/10/21 13:25:14] INFO     [Arturoborrero @ commons.wikimedia.org]: Saving cookies to      wiki.py:141
                             'commons.wikimedia.org_Arturoborrero.pickle'
jcrespo subscribed.Aug 24 2021, 8:58 AM

Hey, @aborrero,

I cannot speak on behalf of the traffic/netops/serviceops team, but given that large files -IIRC- have a different workflow than smaller ones (multi-part upload) and specially given the context of your last comment, I would rise this issue to mw maintainers of the media stack (Structured Commons for the media part and Core Platform for the API part), rather than SREs, as the immediate issue seems to be mediawiki throwing exceptions, based on your logs.

It can later be, of course, an issue on lower layers, but personally I would start reporting all details at that layer first so they can research further why the issue is happening to you.

taavi added a project: Wikimedia-production-error.Aug 24 2021, 9:00 AM
taavi added a project: MediaWiki-Uploading.Aug 25 2021, 2:35 PM
jcrespo moved this task from Backlog to Radar on the SRE board.Aug 25 2021, 5:04 PM
fgiunchedi triaged this task as Medium priority.Aug 30 2021, 8:07 AM
Krinkle removed a project: Wikimedia-production-error.Sep 1 2021, 3:10 AM
aborrero removed projects: Traffic, serviceops.Sep 6 2021, 8:56 AM
Inductiveload subscribed.Sep 14 2021, 5:06 PM

The DBQueryError bit sounds like T278389

Fastily subscribed.Oct 6 2021, 6:11 AM
aborrero added a parent task: T292954: [epic] large file uploads to commons.Oct 11 2021, 10:10 AM

This may be very similar to T278389: File upload to Commons via UploadWizard fails with "Caught exception of type Wikimedia\Rdbms\DBQueryError"

Xover subscribed.Oct 31 2021, 11:02 AM

@aborrero Did you specify a -chunked to pwb.py, and if so what (5MB perhaps?)? And did you give it -async?

The bursty upload could be consistent with pwb uploading a ~5MB chunk to the api servers, and then waiting synchronously for it to complete processing (including sending it to Swift with really bad throughput ala T275752) for ~20 seconds, and then uploading the next chunk. The eventual failure DBQueryError is just a generic error that, aiui, might just be that the whole thing took so long that a DB transaction was held open past its hard limit (300s?) and was forcibly terminated.

Given the fix in T275752 it might be worth retrying this upload.

aborrero closed this task as Resolved.Dec 2 2021, 3:49 PM
aborrero claimed this task.

For the record, I was unable to upload the video using the commons web form: https://commons.wikimedia.org/wiki/File:Dactylopterus_Volitans.webm

It took about 1h to upload, but the file is there. Will close the task.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL