internal_api_error_DBQueryError while uploading big files with chunked upload
Closed, DuplicatePublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Yann
	Apr 23 2023, 3:03 PM

Description

Hi, I am not sure if this is new or old, but I get the error

01566: 62/62> Connection seems to be okay. Waiting one more time...
01566: FAILED: internal_api_error_DBQueryError: [cc99b243-1240-4a9f-b789-7f8d992ac444] Caught exception of type Wikimedia\Rdbms\DBQueryError

while uploading a 600 MB file on Wikisource with https://commons.wikimedia.org/wiki/User:Rillke/bigChunkedUpload.js
https://en.wikisource.org/wiki/File:Land_Values,_Vol._10_-_1904-1906.pdf
I wonder if it is not related to T212101

Chrome console

index.php?action=raw&ctype=text/javascript&title=User:Rillke/MwJSBot.js&_=2:1136 mwbot> internal_api_error_DBQueryError: [cc99b243-1240-4a9f-b789-7f8d992ac444] Caught exception of type Wikimedia\Rdbms\DBQueryError  {"error":{"code":"internal_api_error_DBQueryError","info":"[cc99b243-1240-4a9f-b789-7f8d992ac444] Caught exception of type Wikimedia\\Rdbms\\DBQueryError","errorclass":"Wikimedia\\Rdbms\\DBQueryError"},"servedby":"mw2437"}

Nothing in https://commons.wikimedia.org/wiki/Special:UploadStash

Related Objects

Mentioned Here: T340917: Server error 500 after uploading chunk
T278104: Unable to upload to Commons: uploadstash-file-not-found: Key "187kyl5ozj74.xtav8j.51508.djvu" not found in stash
T212101: Lock wait timeout exceeded in UploadFromChunks::updateChunkStatus

Event Timeline

Yann created this task.Apr 23 2023, 3:03 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 23 2023, 3:03 PM

Yann changed the subtype of this task from "Task" to "Production Error".Apr 23 2023, 3:03 PM

Yann updated the task description. (Show Details)

Several attempts, and always some issues.

Now with the same file, I got

01576: 30/31> Chunk uploaded
01576: 31/31> in progress Upload: 100%
01637: 31/31> upload is stuck
01637: 31/31> Connection seems to be okay. Waiting one more time...
01674: 31/31> Server error 504 after uploading chunk: 
Response: upstream request timeout
01674: 31/31> Connection seems to be okay. Waiting one more time... Upload: 100%
01704: 31/31> upload in progress Upload: 100%
01732: FAILED: uploadstash-file-not-found: Key "1a0l9uot9xv8.vpktuh.7.pdf" not found in stash.

Chrome console:

Failed to load resource: the server responded with a status of 504 ()
index.php?action=raw&ctype=text/javascript&title=User:Rillke/MwJSBot.js&_=2:1136 mwbot> uploadstash-file-not-found: Key "1a0l9uot9xv8.vpktuh.7.pdf" not found in stash.  {"error":{"code":"uploadstash-file-not-found","info":"Key \"1a0l9uot9xv8.vpktuh.7.pdf\" not found in stash.","*":"See https://en.wikisource.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes."},"servedby":"mw2435"}

The error message uploadstash-file-not-found is not the same as the db error from the task description. uploadstash-file-not-found is a handled problem that nothing was found in the database, while the db error indicates other problems (like the linked deadlock), so that part sounds more like T278104

Krinkle added projects: MediaWiki-Uploading, Unstewarded-production-error.Apr 27 2023, 3:32 PM

Krinkle moved this task from Untriaged to Apr 2023 on the Wikimedia-production-error board.

The errors I get are the same as in https://phabricator.wikimedia.org/T278104#6935458
Today I succeeded uploading big PDF files (~500 MB) to Commons, so this is really weird.

Now I get

01691: 61/62> Chunk uploaded
01691: 62/62> in progress Upload: 100%
01767: 62/62> Server error 504 after uploading chunk: 
Response: upstream request timeout
01767: 62/62> upload in progress Upload: 100%
01796: FAILED: internal_api_error_DBQueryError: [802fb2ec-4ded-43c5-a737-f78cfc383742] Caught exception of type Wikimedia\Rdbms\DBQueryError

Chrome console:

checked runtime.lastError: The message port closed before a response was received.
...
index.php?action=raw&ctype=text/javascript&title=User:Rillke/MwJSBot.js&_=2:1111 mwbot> Hello. I am your MwJSBot framework.
/w/api.php:1     Failed to load resource: the server responded with a status of 504 ()
index.php?action=raw&ctype=text/javascript&title=User:Rillke/MwJSBot.js&_=2:1136 mwbot> internal_api_error_DBQueryError: [802fb2ec-4ded-43c5-a737-f78cfc383742] Caught exception of type Wikimedia\Rdbms\DBQueryError  {"error":{"code":"internal_api_error_DBQueryError","info":"[802fb2ec-4ded-43c5-a737-f78cfc383742] Caught exception of type Wikimedia\\Rdbms\\DBQueryError","errorclass":"Wikimedia\\Rdbms\\DBQueryError"},"servedby":"mw1493"}
fail @ index.php?action=raw&ctype=text/javascript&title=User:Rillke/MwJSBot.js&_=2:1136
DevTools failed to load source map: Could not load content for chrome-extension://gighmmpiobklfepjocnamgkkbiglidom/browser-polyfill.js.map: System error: net::ERR_FILE_NOT_FOUND
File:Land_Values,_Vol._09_-_1902-1904.pdf:1 The resource https://en.wikisource.org/static/images/project-logos/enwikisource.png was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate `as` value and it is preloaded intentionally.

https://en.wikisource.org/wiki/File:Land_Values,_Vol._09_-_1902-1904.pdf
Just in case you want to try, the file is available online at https://www.henrygeorgefoundation.org/publications/land-liberty-archive/1900s.html

In T335243#8812060, @Yann wrote:

Now I get

01691: 61/62> Chunk uploaded
01691: 62/62> in progress Upload: 100%
01767: 62/62> Server error 504 after uploading chunk: 
Response: upstream request timeout
01767: 62/62> upload in progress Upload: 100%
01796: FAILED: internal_api_error_DBQueryError: [802fb2ec-4ded-43c5-a737-f78cfc383742] Caught exception of type Wikimedia\Rdbms\DBQueryError

802fb2ec-4ded-43c5-a737-f78cfc383742 is:

Error 1205: Lock wait timeout exceeded; try restarting transaction
Function: UploadFromChunks::updateChunkStatus
Query: UPDATE  `uploadstash` SET us_status = 'chunks',us_chunk_inx = 61,us_size = 639350005 WHERE us_key = '1a0s58n72yso.jurfph.7.pdf'

… i.e. this is T212101: Lock wait timeout exceeded in UploadFromChunks::updateChunkStatus. :-(

I successfully uploaded several ~1 GB PDF files yesterday, but https://commons.wikimedia.org/wiki/File:Land_Values,_Vol._09_-_1902-1904.pdf still fails with the same error, also on Commons as in Wikisource.

00501: 20/62> in progress Upload: 100%
00531: 20/62> Chunk uploaded
00531: 21/62> in progress Upload: 100%
00558: FAILED: internal_api_error_UploadChunkFileException: [5df82564-e56a-421a-9704-fd7c97890994] Caught exception of type UploadChunkFileException

Chrome console:

index.php?action=raw&ctype=text/javascript&title=User:Rillke/MwJSBot.js&_=2:1136 mwbot> internal_api_error_UploadChunkFileException: [5df82564-e56a-421a-9704-fd7c97890994] Caught exception of type UploadChunkFileException  {"error":{"code":"internal_api_error_UploadChunkFileException","info":"[5df82564-e56a-421a-9704-fd7c97890994] Caught exception of type UploadChunkFileException","errorclass":"UploadChunkFileException"},"servedby":"mw1428"}
fail @ index.php?action=raw&ctype=text/javascript&title=User:Rillke/MwJSBot.js&_=2:1136
DevTools failed to load source map: Could not load content for chrome-extension://gighmmpiobklfepjocnamgkkbiglidom/browser-polyfill.js.map: System error: net::ERR_FILE_NOT_FOUND

May be not directly related, but T340917 happens in the same sutuation.

Umherirrender unsubscribed.Nov 29 2023, 10:37 PM

T212101 and T278104 were both detailed in this report, but it seems most likely that this was T212101

TheDJ closed this task as a duplicate of T212101: Lock wait timeout exceeded in UploadFromChunks::updateChunkStatus.Dec 23 2023, 7:36 PM

internal_api_error_DBQueryError while uploading big files with chunked upload Closed, DuplicatePublicPRODUCTION ERRORActions

Description

Related Objects

Event Timeline

internal_api_error_DBQueryError while uploading big files with chunked upload
Closed, DuplicatePublicPRODUCTION ERROR
Actions