Page MenuHomePhabricator

File upload to Commons via UploadWizard fails with "Caught exception of type Wikimedia\Rdbms\DBQueryError"
Open, Needs TriagePublicPRODUCTION ERROR

Description

When trying to upload the (CC BY 4.0) PDF from here to Commons, the progress bar makes it until about 95%, and then I am getting errors of the type

[6813f71f-3560-48b8-b071-deda1b801f18] Caught exception of type Wikimedia\Rdbms\DBQueryError

that result in the upload being interrupted.

.

A search for the error message on Phabricator yielded T229605, which is marked of a duplicate of T229589, which is marked as resolved, so I think a new ticket is needed.

Details

Stack Trace

from /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1700)
#0 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1678): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1244): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(2479): Wikimedia\Rdbms\Database->query(string, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update(string, array, string, string)
#4 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#5 /srv/mediawiki/php-1.36.0-wmf.35/includes/upload/UploadFromChunks.php(289): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string)
#6 /srv/mediawiki/php-1.36.0-wmf.35/includes/upload/UploadFromChunks.php(263): UploadFromChunks->updateChunkStatus()
#7 /srv/mediawiki/php-1.36.0-wmf.35/includes/api/ApiUpload.php(251): UploadFromChunks->addChunk(string, integer, integer)
#8 /srv/mediawiki/php-1.36.0-wmf.35/includes/api/ApiUpload.php(130): ApiUpload->getChunkResult(array)
#9 /srv/mediawiki/php-1.36.0-wmf.35/includes/api/ApiUpload.php(101): ApiUpload->getContextResult()
#10 /srv/mediawiki/php-1.36.0-wmf.35/includes/api/ApiMain.php(1646): ApiUpload->execute()
#11 /srv/mediawiki/php-1.36.0-wmf.35/includes/api/ApiMain.php(616): ApiMain->executeAction()
#12 /srv/mediawiki/php-1.36.0-wmf.35/includes/api/ApiMain.php(587): ApiMain->executeActionWithErrorHandling()
#13 /srv/mediawiki/php-1.36.0-wmf.35/api.php(90): ApiMain->execute()
#14 /srv/mediawiki/php-1.36.0-wmf.35/api.php(45): wfApiMain()
#15 /srv/mediawiki/w/api.php(3): require(string)
#16 {main}

Event Timeline

Peachey88 changed the subtype of this task from "Task" to "Production Error".Mar 24 2021, 11:37 PM

The update is using us_key which is an unique key. It is not the primary key, but there is only one row.
Maybe this needs a DBA to find the other transaction which holds the row.

A possible issue is when the chunks are uploaded in parallel.
But the next chunk could only be uploaded if the current chunk gets a success from the api.

00591: 61/63> in progress Upload: 98%
00601: 61/63> Chunk uploaded
00601: 62/63> in progress Upload: 99%
00610: 62/63> Chunk uploaded
00610: 63/63> in progress Upload: 95%
00620: 63/63> upload is stuck
00620: 63/63> Connection seems to be okay. Waiting one more time...
00625: 63/63> upload is stuck
00625: 63/63> Connection seems to be okay. Waiting one more time...
00630: 63/63> upload is stuck
00630: 63/63> Connection seems to be okay. Waiting one more time...
00635: 63/63> upload is stuck
00635: 63/63> Connection seems to be okay. Waiting one more time...
00640: 63/63> upload is stuck
00640: 63/63> Connection seems to be okay. Waiting one more time...
00645: 63/63> upload is stuck
00645: 63/63> Connection seems to be okay. Waiting one more time...
00650: 63/63> upload is stuck
00650: 63/63> Connection seems to be okay. Waiting one more time...
00655: 63/63> upload is stuck
00655: 63/63> Connection seems to be okay. Waiting one more time...
00660: 63/63> upload is stuck
00660: 63/63> Connection seems to be okay. Waiting one more time...
00665: 63/63> upload is stuck
00665: 63/63> Connection seems to be okay. Waiting one more time...
00670: 63/63> upload is stuck
00670: 63/63> Server error 0 after uploading chunk:
Response:
00670: 63/63> Connection seems to be okay. Re-sending this request.
00670: 63/63> Connection seems to be okay. Re-sending this request. Upload: 100%
00680: 63/63> upload is stuck
00680: 63/63> Connection seems to be okay. Waiting one more time...
00685: 63/63> upload is stuck
00685: 63/63> Connection seems to be okay. Waiting one more time...
00689: FAILED: internal_api_error_DBQueryError: [11111111-11a1-1a1a-11a1-a1aa111aa1a1] Caught exception of type Wikimedia\Rdbms\DBQueryError

I censored the code as I don't know what it stands for. Tried to overwrite https://en.wikipedia.org/wiki/File:Jefferson_Davis_High_School_band_2020.ogv with some better quality. I'll try again but I have a feeling I'll just have to downgrade the quality.

Edit: tried again, same Wikimedia\Rdbms\DBQueryError. Different code (the thing I censored) but this code is probably irrelevant anyway. Trying again now without stash and async.

00598: 61/63> in progress Upload: 99%
00607: 61/63> Chunk uploaded
00607: 62/63> in progress Upload: 100%
00617: 62/63> Chunk uploaded
00617: 63/63> in progress Upload: 100%
00684: 63/63> Server error 504 after uploading chunk: Gateway Timeout
Response: upstream request timeout
00684: 63/63> upload in progress Upload: 100%
00704: FAILED: internal_api_error_DBQueryError: [(censored)] Caught exception of type Wikimedia\Rdbms\DBQueryError

No luck without stash and async. I'll just lower the bitrate. It's visibly worse. Not completely terrible, but visible.

First time I try to upload the <100MiB file I get:

Service Temporarily Unavailable

Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes.

Second time I get:

Could not acquire lock for "mwstore://local-multiwrite/local-public/7/78/Jefferson_Davis_High_School_band_2020.ogv".
Return to Main Page.

Maybe the limit is indeed 100MB and not 100MiB. Shouldn't I get a different error message though? I'll try shaving a few MB off.

Edit: Still can't overwrite, even under 100MB. Alternates between the acquire lock and experiencing a technical problem errors.

Request from (my ip) via cp3052 frontend, Varnish XID 690378672
Error: 503, Backend fetch failed at Thu, 22 Apr 2021 16:35:19 GMT

There are several uploads to Commons using video2commons like https://commons.wikimedia.org/wiki/File:Fire_transport_breaks_through_to_the_fire_20210422_211433.webm that are over 100MB. @Fae just uploaded https://commons.wikimedia.org/wiki/File:Iacobi_Berengari_Carpensis_..._De_fractura_cranii_liber_aureus._Hactenus_desideratus_-_(Jacopo_Berengario_da_Carpi)_(IA_hin-wel-all-00002134-001).pdf which is slightly over 100MB. (IA Query "collection:(additional_collections) date:[1000 TO 1869] " hin-wel-all-00002134-001 Category:Old books in Internet Archive additional collections (COM:IA books#query) (1629 #87533))

I don't know how the upload process for video2commons or IA Query works. Are they different from Special:Upload, UploadWizard and PyWikiBot which all seem to consistently fail to upload anything over 100MB?