Lock wait timeout exceeded in UploadFromChunks::updateChunkStatus (for retry of last chunk after http timeout)
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	Yann
	Dec 17 2018, 7:38 AM

Project Tags

Referenced Files

	F27904933: error.png
	Jan 16 2019, 2:02 AM

Subscribers

View All 12 Subscribers

Description

After T200820#4826332 I tried again without "stash and async" (as recommended on https://commons.wikimedia.org/wiki/User_talk:Rillke/bigChunkedUpload.js#Troubleshooting ), and I got the following

03054: 287/287> Server error 503 after uploading chunk: 
Response: <!DOCTYPE html>
<html lang="en" dir="ltr">
<meta charset="utf-8">
<title>Wikimedia Error</title>
<style>
* { margin: 0; padding: 0; }
body { background: #fff; font: 15px/1.6 sans-serif; color: #333; }
.content { margin: 7% auto 0; padding: 2em 1em 1em; max-width: 640px; }
img { float: left; margin: 0 2em 2em 0; }
a img { border: 0; }
h1 { margin-top: 1em; font-size: 1.2em; }
p { margin: 0.7em 0 1em 0; }
a { color: #0645AD; text-decoration: none; }
a:hover { text-decoration: underline; }
</style>
03054: 287/287> upload in progress Upload: 100%
03078: FAILED: internal_api_error_DBQueryError: [XBZngQpAAEwAACGSPNYAAACO] Caught exception of type Wikimedia\Rdbms\DBQueryError

Is this the same issue?
File in stash is 1698l1a6s3kw.c4ox7h.1.pdf

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Upload: For sync chunk uploads set assembling stage in session cache	Umherirrender	mediawiki/core	master	+63 -21

Customize query in gerrit

Related Objects

Mentioned In: T386640: "Could not acquire lock" error when publishing larger files
T391473: Common API upload failures with very large files
T335243: internal_api_error_DBQueryError while uploading big files with chunked upload
T38587: Fatal error when uploading a file to Commons (UploadStashFileNotFoundException)
Mentioned Here: T274589: No atomic section is open (got LocalFile::lockingTransaction)
T283045: LocalFile::lock() rearchitecting
T427949: Uncompressed TIFFs on commons
T391473: Common API upload failures with very large files
T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0".

Event Timeline

Yann created this task.Dec 17 2018, 7:38 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 17 2018, 7:38 AM

Yann mentioned this in T38587: Fatal error when uploading a file to Commons (UploadStashFileNotFoundException).Dec 17 2018, 7:38 AM

[XBZngQpAAEwAACGSPNYAAACO] /w/api.php   Wikimedia\Rdbms\DBQueryError from line 1506 of /srv/mediawiki/php-1.33.0-wmf.8/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: UPDATE  `uploadstash` SET us_status = 'chunks',us_chunk_inx = '286',us_size = '1201531325' WHERE us_key = '1698l1a6s3kw.c4ox7h.1.pdf'
Function: UploadFromChunks::updateChunkStatus
Error: 1205 Lock wait timeout exceeded; try restarting transaction (10.64.48.23)
#0 /srv/mediawiki/php-1.33.0-wmf.8/includes/libs/rdbms/database/Database.php(1476): Wikimedia\Rdbms\Database->makeQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.33.0-wmf.8/includes/libs/rdbms/database/Database.php(1236): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /srv/mediawiki/php-1.33.0-wmf.8/includes/libs/rdbms/database/Database.php(2114): Wikimedia\Rdbms\Database->query(string, string)
#3 /srv/mediawiki/php-1.33.0-wmf.8/includes/upload/UploadFromChunks.php(300): Wikimedia\Rdbms\Database->update(string, array, array, string)
#4 /srv/mediawiki/php-1.33.0-wmf.8/includes/upload/UploadFromChunks.php(273): UploadFromChunks->updateChunkStatus()
#5 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiUpload.php(226): UploadFromChunks->addChunk(string, integer, integer)
#6 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiUpload.php(132): ApiUpload->getChunkResult(array)
#7 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiUpload.php(104): ApiUpload->getContextResult()
#8 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiMain.php(1595): ApiUpload->execute()
#9 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiMain.php(531): ApiMain->executeAction()
#10 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#11 /srv/mediawiki/php-1.33.0-wmf.8/api.php(87): ApiMain->execute()
#12 /srv/mediawiki/w/api.php(3): include(string)
#13 {main}

So there are two separate processes trying to update the same upload status, and one of the locks the row and the other times out.
Not sure how that can happen... maybe the tool does parallel upload of multiple chunks at the same time, but even so, the status update should only happen when the upload of that chunk has finished, so the row should not be locked for long, and a lock timeout takes multiple seconds to happen.

And this only happens on the last chunk, when trying to build the whole file, which is the same as the other 2 bugs related. I don't think that a coincidence.

Aklapper added a project: MediaWiki-Uploading.Dec 17 2018, 10:35 AM

Restricted Application added a project: Multimedia. · View Herald TranscriptDec 17 2018, 10:35 AM

Tgr renamed this task from internal_api_error_DBQueryError to Lock wait timeout exceeded in UploadFromChunks::updateChunkStatus.Dec 17 2018, 10:50 AM

For the record, a 472 MB file got through: https://commons.wikimedia.org/wiki/File:M%C3%A9tin_-_L%E2%80%99Inde_d%E2%80%99aujourd%E2%80%99hui%2C_%C3%A9tude_sociale%2C_1918.pdf
while a 903 MB file failed: https://commons.wikimedia.org/wiki/File:L%27Illustration,_Sep-Dec_1849.pdf

I could upload the 3 files which caused this bug.

This happened again after uploading a 1.9 GB file. :((

04283: 458/458> in progress Upload: 100%
04411: 458/458> Server error 503 after uploading chunk: 
Response: <!DOCTYPE html>
<html lang="en" dir="ltr">
<meta charset="utf-8">
<title>Wikimedia Error</title>
<style>
* { margin: 0; padding: 0; }
body { background: #fff; font: 15px/1.6 sans-serif; color: #333; }
.content { margin: 7% auto 0; padding: 2em 1em 1em; max-width: 640px; }
img { float: left; margin: 0 2em 2em 0; }
a img { border: 0; }
h1 { margin-top: 1em; font-size: 1.2em; }
p { margin: 0.7em 0 1em 0; }
a { color: #0645AD; text-decoration: none; }
a:hover { text-decoration: underline; }
</style>
04411: 458/458> upload in progress Upload: 99%
04431: FAILED: internal_api_error_DBQueryError: [XD4dAwpAIDYAAB0dQIIAAACY] Caught exception of type Wikimedia\Rdbms\DBQueryError

https://commons.wikimedia.org/wiki/File:L%27Illustration,_v.20,_Jul-Dec_1852.pdf
See stash
16bsomjwzu4s.u4y885.1.pdf
16bss6f5rz7g.617yma.1.pdf

Same error, lock wait timeout in UploadFromChunks::updateChunkStatus (at chunk 457). Maybe we should add some debug logging for when exactly each chunk starts / ends uploading, to see if the client does anything weird there.

Wilfredor subscribed.Jan 16 2019, 12:35 AM

This comment was removed by Wilfredor.

Wilfredor added a comment.Jan 16 2019, 1:57 AM

This comment was removed by Wilfredor.

Same problem, I cant upload my images because this problem, also with Uploadwizard with images TIFF with more 600Mb of size (TIFF with internal zip compression).

Failed to load resource: the server responded with a status of 404 () https://commons.wikimedia.org/wiki/Special:UploadStash/thumb/16btxukj8dc8.3faxuw.32521.tif/lossy-page1-120px-16btxukj8dc8.3faxuw.32521.tif.jpg?0.6459894966662545

The sample file that Im trying to upload is:
https://archive.org/download/SacreCoeurDeJesusByLouisJobin1878/Sacre%20Coeur%20de%20Jesus%20by%20Louis%20Jobin%201878.tif

Also the follow error while uploading via URL:

identify command failed: '/usr/bin/identify' -format "[BEGIN]page=%p\nalpha=%A\nalpha2=%r\nheight=%h\nwidth=%w\ndepth=%z[END]" '/tmp/URLb7638a2c66b1.urlupload_' 2>&1

executing the above command locally I get:

$ '/usr/bin/identify' -format "[BEGIN]page=%p\nalpha=%A\nalpha2=%r\nheight=%h\nwidth=%w\ndepth=%z[END]" Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif
identify: Incompatible type for "RichTIFFIPTC"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/912.
identify: Unknown field with tag 34864 (0x8830) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/912.
identify: Incompatible type for "FileSource"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/912.
identify: Incompatible type for "SceneType"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/912.
identify: Wrong data type 3 for "GainControl"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/912.
identify: Unknown field with tag 42033 (0xa431) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/912.
identify: Unknown field with tag 42034 (0xa432) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/912.
identify: Unknown field with tag 42036 (0xa434) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/912.
identify: Incompatible type for "RichTIFFIPTC"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/912.
identify: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
identify: cache resources exhausted `Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif' @ error/cache.c/OpenPixelCache/3945.
identify: memory allocation failed `Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif' @ error/psd.c/ReadPSDLayers/1737.
[BEGIN]page=0
alpha=False
alpha2=DirectClass sRGB
height=8912
width=6160
$ identify --version
Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP
Delegates (built-in): bzlib djvu fftw fontconfig freetype jbig jng jp2 jpeg lcms lqr ltdl lzma openexr pangocairo png tiff wmf x xml zlib

Out of memory?
Any hints what else can we do here?

The above was on a machine with 8GB RAM + 8GB swap. On a machine with 100GB RAM I get:

$ '/usr/bin/identify' -format "[BEGIN]page=%p\nalpha=%A\nalpha2=%r\nheight=%h\nwidth=%w\ndepth=%z[END]" Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif
identify: Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif: wrong data type 7 for "RichTIFFIPTC"; tag ignored. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
identify: Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif: unknown field with tag 34864 (0x8830) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/768.
identify: Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif: wrong data type 3 for "GainControl"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/768.
identify: Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif: unknown field with tag 42033 (0xa431) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/768.
identify: Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif: unknown field with tag 42034 (0xa432) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/768.
identify: Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif: unknown field with tag 42036 (0xa434) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/768.
identify: Sacre_Coeur_de_Jesus_by_Louis_Jobin_1878.tif: wrong data type 7 for "RichTIFFIPTC"; tag ignored. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
[BEGIN]page=0
alpha=False
alpha2=DirectClass sRGB
height=8912
width=6160
depth=16[END]
$ echo $?
0
$ identify --version
Version: ImageMagick 6.7.7-10 2014-03-08 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP

So this is likely OOM problem.

Let's keep this task to be about the lock wait timeout error, please.

Still seen as of 1.37.0-wmf.1, happening at the rate of a few 10s of times per hour (max of 30ish)

mwversion: 1.37.0-wmf.1
reqId: d8c15ab4-c638-4b2a-996c-51d9f09841dd
Find reqId in Logstash
Find normalized_message in Logstash

normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1205: Lock wait timeout exceeded; try restarting transaction (10.64.48.124)
Function: UploadFromChunks::updateChunkStatus
Query: UPDATE  `uploadstash` SET us_status = 'chunks',us_chunk_inx =

exception.trace

from /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(1738)
#0 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(1722): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(1697): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(1260): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(2498): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update(string, array, string, string)
#5 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.37.0-wmf.1/includes/upload/UploadFromChunks.php(289): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string)
#7 /srv/mediawiki/php-1.37.0-wmf.1/includes/upload/UploadFromChunks.php(263): UploadFromChunks->updateChunkStatus()
#8 /srv/mediawiki/php-1.37.0-wmf.1/includes/api/ApiUpload.php(251): UploadFromChunks->addChunk(string, integer, integer)
#9 /srv/mediawiki/php-1.37.0-wmf.1/includes/api/ApiUpload.php(130): ApiUpload->getChunkResult(array)
#10 /srv/mediawiki/php-1.37.0-wmf.1/includes/api/ApiUpload.php(101): ApiUpload->getContextResult()
#11 /srv/mediawiki/php-1.37.0-wmf.1/includes/api/ApiMain.php(1652): ApiUpload->execute()
#12 /srv/mediawiki/php-1.37.0-wmf.1/includes/api/ApiMain.php(622): ApiMain->executeAction()
#13 /srv/mediawiki/php-1.37.0-wmf.1/includes/api/ApiMain.php(593): ApiMain->executeActionWithErrorHandling()
#14 /srv/mediawiki/php-1.37.0-wmf.1/api.php(90): ApiMain->execute()
#15 /srv/mediawiki/php-1.37.0-wmf.1/api.php(45): wfApiMain()
#16 /srv/mediawiki/w/api.php(3): require(string)
#17 {main}

Restricted Application added a project: Structured-Data-Backlog. · View Herald TranscriptApr 28 2021, 6:23 PM

thcipriani added a project: Wikimedia-production-error.Apr 28 2021, 6:24 PM

thcipriani moved this task from Untriaged to Older on the Wikimedia-production-error board.

There is logging in the code (UploadFromChunks::updateChunkStatus), not sure if that can be found for this request:

		wfDebug( __METHOD__ . " update chunk status for {$this->mFileKey} offset:" .
			$this->getOffset() . ' inx:' . $this->getChunkIndex() );

The update on the table is done using uploadstash.us_key, which is not the primary key, but a unique index, which should be fast as the primary key and not resulting in deadlocks.

CBogen edited projects, added: Structured-Data-Backlog (Current Work); removed: Structured-Data-Backlog.May 12 2021, 2:15 PM

CBogen moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.

Matlin subscribed.Jun 28 2021, 12:21 PM

CBogen edited projects, added: Structured-Data-Backlog; removed: Structured-Data-Backlog (Current Work).Jun 30 2021, 4:54 PM

CBogen moved this task from Triage to Multimedia on the Structured-Data-Backlog board.

Umherirrender unsubscribed.Nov 25 2021, 6:55 PM

Yann mentioned this in T335243: internal_api_error_DBQueryError while uploading big files with chunked upload .Apr 23 2023, 3:03 PM

I have removed a log filter from Kibana MediaWiki New Errors dashboard which pointed back to this task. There is no hit over the last eight weeks.

The filter was T212101 Lock wait timeout exceeded in UploadFromChunks::updateChunkStatus

{
  "query": {
    "match_phrase": {
      "error.message": "Function: UploadFromChunks::updateChunkStatus"
    }
  }
}

TheDJ merged a task: T335243: internal_api_error_DBQueryError while uploading big files with chunked upload .Dec 23 2023, 7:36 PM

TheDJ added subscribers: TheDJ, Jdforrester-WMF.

Umherirrender mentioned this in T391473: Common API upload failures with very large files.Apr 9 2025, 9:09 PM

I'm currently poking at this via T391473; the error definitely still occurs!

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptApr 9 2025, 9:13 PM

I spent some time on this since it's blocking me compressing tiff files (T427949). I put result of my investigation so far before calling it a day and coming back to stare at it more tomorrow.

So I run a python script to reupload a file with chunks using pywikibot. The files I'm reuploading are obscure so I am sure nothing is writing the same row (and it happens consistently). Here is the around 300 of the errors to investigate: https://logstash.wikimedia.org/goto/e5c09582d7c1b27a6d03b56dcbd1e319

The error is this:

Error 1205 from MediaWiki\Upload\UploadFromChunks::updateChunkStatus, Lock wait timeout exceeded; try restarting transaction UPDATE  `uploadstash` SET us_status = 'chunks',us_chunk_inx = 30,us_size = 960105420 WHERE us_key = '1cumins7kwkw.9xoht5.2158737.tif' db1160

It can't grab the lock for us_key key. As I said, I highly doubt anyone else is uploading the same files consistently the same time as I do. So it is some lock from the previous upload chunk request that is somehow not being release (in case it's not clear, chunked upload is just a series of API requests back to back. I put some sleep between the requests and still no impact). The requests are serial so it can't be my bot is interfering with itself naturally (specially since this is a problem in js too).

Another weird thing is that since pywikibot has a retry mechanism, it tries again after 10s and 20s and still can't grab the lock. I don't think we have anything that would lock the row for that long. It also waits for 15s each time (https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2026.08.06?id=7uFy2Z8BYIDqwC7cXBnh). So even after one minute the lock on the key is not released. I have looked at "TRANSACTIONS" section of HOW ENGINE INNODB STATUS\G in db1160 (the master db of commons) and no active transaction seems to be holding the lock (but I might be something obvious)

One random observation: When I picked 5MB for the chunk size, in many many cases it started to fail at around 300th chunk (1.5GB). When I bumped it to 30MB (assuming the time spent between requests will be larger and so the chance of lock contention would be lower). It started to fail at around 30th chunk (roughly 900MB). I reduced it to 1MB, maybe it'll be able to upload the files finally.

Here is the run errors:

(p3) ladsgroup@runner:~$ python3 tiff_compression2.py  >> run_log2
Logging in to commons:commons as Dexbot@bot_name
ERROR: Traceback (most recent call last):
  File "/home/ladsgroup/p3/lib/python3.13/site-packages/pywikibot/data/api/_requests.py", line 733, in _http_request
    response = http.request(self.site, uri=uri,
                            method='GET' if use_get else 'POST',
                            data=data, headers=headers)
  File "/home/ladsgroup/p3/lib/python3.13/site-packages/pywikibot/comms/http.py", line 328, in request
    r = fetch(baseuri, headers=headers, **kwargs)
  File "/home/ladsgroup/p3/lib/python3.13/site-packages/pywikibot/comms/http.py", line 505, in fetch
    callback(response)
    ~~~~~~~~^^^^^^^^^^
  File "/home/ladsgroup/p3/lib/python3.13/site-packages/pywikibot/comms/http.py", line 378, in error_handling_callback
    raise ServerError(response)
pywikibot.exceptions.ServerError: HTTPSConnectionPool(host='commons.wikimedia.org', port=443): Read timed out. (read timeout=45)

WARNING: Waiting 5.0 seconds before retrying.
WARNING: API error internal_api_error_DBQueryError: [702a2061-f7d9-415f-8326-56a21f70f883] Caught exception of type Wikimedia\Rdbms\DBQueryError
ERROR: Detected MediaWiki API exception internal_api_error_DBQueryError: [702a2061-f7d9-415f-8326-56a21f70f883] Caught exception of type Wikimedia\Rdbms\DBQueryError
[errorclass: Wikimedia\Rdbms\DBQueryError;
 servedby: mw-api-ext.eqiad.main-7d47b7d59c-cqxf2]; retrying
WARNING: Waiting 10.0 seconds before retrying.
WARNING: API error internal_api_error_DBQueryError: [7e8beb78-b424-4abc-b86f-ff71b223de82] Caught exception of type Wikimedia\Rdbms\DBQueryError
ERROR: Detected MediaWiki API exception internal_api_error_DBQueryError: [7e8beb78-b424-4abc-b86f-ff71b223de82] Caught exception of type Wikimedia\Rdbms\DBQueryError
[errorclass: Wikimedia\Rdbms\DBQueryError;
 servedby: mw-api-ext.eqiad.main-7d47b7d59c-fqmn9]; retrying
WARNING: Waiting 20.0 seconds before retrying.
WARNING: API error stashfailed: No chunked upload session with this key.

My leading hypothesis is that one upload request gets closed by the client because it's too slow, the server just doesn't close the transaction but ends the session without rolling back for a while. When pywikibot/client tries to retry, it basically ends up in a lock contention with the ghost of request past.

That's probably it. ApiUpload sets useTransactionalTimeLimit, which sets ignore_user_abort

It makes sense in hindsight. we try to protect against multi write inconsistencies for when a connection gets killed, but with uploadchunks we have a sequence of connections, with a lock connecting that sequence. So when one request times out at the http level or something, then the transaction just keeps running for max wgTransactionalTimeLimit. If the next upload chunk then starts, the lock hasn't been released yet.

TheDJ reopened this task as Open.Aug 7 2026, 9:14 AM

Ankry unsubscribed.Aug 7 2026, 10:20 AM

@GPSLeo btw. I think this is why chunked uploads were failing. I try to work on it.

Ladsgroup edited projects, added: MediaWiki-Media-Platform-Team; removed: Structured-Data-Backlog, Structured Data Engineering.Aug 7 2026, 10:56 AM

Ladsgroup added a project: Essential-Work.

Ladsgroup mentioned this in T386640: "Could not acquire lock" error when publishing larger files.Aug 7 2026, 11:42 AM

Logstash only has FileOperations logs at the WARNING and ERROR level, but we have DEBUG level logs on mwlog1003.

Taking the example of the error at 2026-07-11 16:39:22.

There are 332 chunks in the file, and the PUT requests for all chunk uploads all complete promptly. The final chunk request takes on the role of assembly and begins fetching chunks at 16:37:50 which is done at 16:37:56.

The exec log shows that it runs scripts/retrieveMetaData.sh three times, each time failing with "TIFFReadDirectory: Warning, Unknown field with tag 34735 (0x87af) encountered". The timestamps are not forwarded properly but it probably took about 22 seconds each time.

At 16:39:38 (in between the second and third metadata attempt) it logs "Stashed combined (332 chunks) of 1csdzgor2pts.dr5p5f.2158737.tif under new name 1cse064lkjk0.2jy21n.2158737.tif" although there is no PUT request for the combined file in the FileOperation log.

The main transaction is committed at 16:40:02, with the report:

Suboptimal transaction [commonswiki db1160 TRX#839874c6ae]:
0  0.001s role-primary: UPDATE  `uploadstash` SET us_status = '?',us_chunk_inx = '?',us_size = '?' WHERE us_key = '?'
1  107.460s ...delay...
2  0.001s role-primary: INSERT INTO `uploadstash` (us_user,us_key,us_orig_path,us_path,us_props,us_size,us_sha1,us_mime,us_media_type,us_image_width,us_image_height,us_image_bits,us_source_type,us_timestamp,us_status) VALUES '?'
3  0.000s role-primary: DELETE FROM `uploadstash` WHERE us_key = '?'
4  23.367s ...delay...

Meanwhile, two retries of the final chunk upload are performed, in separate requests at 16:38:40 and 16:39:07. These two requests terminate with a DB lock wait error as in the task description at 16:38:56 and 16:39:22 respectively.

In the assembly request, it deletes chunks in a DeferredUpdate, completing at 16:40:07.

Thoughts:

POST request are supposed to time out after 200s, so whatever layer it is that is timing out and retrying after 50 or 60s needs to stop doing that. Set the client timeout to 240s. This request took about 132s which is slow but shouldn't be fatal.
As I've previously said at e.g. T274589#7091659, concurrent requests to modify the same file should just error out with zero wait. I purportedly fixed it with T283045 but I probably didn't test chunked uploads.
Metadata generation should only happen once.
We have AssembleUploadChunksJob but the API client has to opt in to it with the async parameter. If you're uploading enormous files then it's safer to set that option, so that assembling and publishing the file doesn't get killed with a PHP timeout. We could have a file size limit for non-async uploads. It's certainly better to set it and to poll for success than to do a client-side abort after 60s.

Thanks for the thorough investigation. For my case, I switched to async but let me investigate other issues (and we can probably make upload wizard be async after a certain size too)

JTweed-WMF triaged this task as High priority.Aug 13 2026, 9:15 AM

JTweed-WMF moved this task from Inbox (Untriaged) to Backlog (Triaged) on the MediaWiki-Media-Platform-Team board.

hashar unsubscribed.Aug 18 2026, 6:58 AM

Change #1336136 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Upload: For sync chunk uploads set assembling stage in session cache

https://gerrit.wikimedia.org/r/1336136

gerritbot added a project: Patch-For-Review.Sep 4 2026, 10:29 PM

Change #1336136 merged by jenkins-bot:

[mediawiki/core@master] Upload: For sync chunk uploads set assembling stage in session cache

https://gerrit.wikimedia.org/r/1336136

ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.20; 2026-09-15).Sep 12 2026, 6:00 PM

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2026, 6:30 PM

Umherirrender renamed this task from Lock wait timeout exceeded in UploadFromChunks::updateChunkStatus to Lock wait timeout exceeded in UploadFromChunks::updateChunkStatus (for retry of last chunk after http timeout).Sep 12 2026, 10:36 PM

Umherirrender closed this task as Resolved.

Umherirrender claimed this task.

After deployment of the merged change, the retry of the last chunk after a http timeout should result in the apierror-stashfailed-complete api error and action=upload&checkstatus= must be used to see the state of the assembling for this upload and know the end of the assembling before issue the publish step. It would also show errors, if there is failure for the assembling, like swift error or even the post timeout is not enough. In most cases a second upload with the async flag could help.

Lock wait timeout exceeded in UploadFromChunks::updateChunkStatus (for retry of last chunk after http timeout)Closed, ResolvedPublicPRODUCTION ERRORActions

Description

Details

Related Objects

Event Timeline

Lock wait timeout exceeded in UploadFromChunks::updateChunkStatus (for retry of last chunk after http timeout)
Closed, ResolvedPublicPRODUCTION ERROR
Actions