Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent 14.199902057648 second(s) in writes, exceeding the limit of 3
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	thcipriani
	Nov 4 2021, 3:36 PM

Description

Error

mwversion: 1.38.0-wmf.7
reqId: aea58de5-c4d8-4512-a4f4-84ab2040b8d4
Find reqId in Logstash
Find normalized_message in Logstash

normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent 14.199902057648 second(s) in writes, exceeding the limit of 3

exception.trace

from /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1763)
#0 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2344): Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#1 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1781): Wikimedia\Rdbms\LoadBalancer->forEachOpenPrimaryConnection(Closure)
#2 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/lbfactory/LBFactory.php(249): Wikimedia\Rdbms\LoadBalancer->approvePrimaryChanges(array, string, integer)
#3 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(236): Wikimedia\Rdbms\LBFactory::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#4 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/lbfactory/LBFactory.php(251): Wikimedia\Rdbms\LBFactoryMulti->forEachLB(Closure, array)
#5 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/lbfactory/LBFactory.php(315): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#6 /srv/mediawiki/php-1.38.0-wmf.7/includes/MediaWiki.php(684): Wikimedia\Rdbms\LBFactory->commitPrimaryChanges(string, array)
#7 /srv/mediawiki/php-1.38.0-wmf.7/includes/api/ApiMain.php(893): MediaWiki::preOutputCommit(DerivativeContext)
#8 /srv/mediawiki/php-1.38.0-wmf.7/includes/api/ApiMain.php(838): ApiMain->executeActionWithErrorHandling()
#9 /srv/mediawiki/php-1.38.0-wmf.7/api.php(90): ApiMain->execute()
#10 /srv/mediawiki/php-1.38.0-wmf.7/api.php(45): wfApiMain()
#11 /srv/mediawiki/w/api.php(3): require(string)
#12 {main}

Impact

Notes

All happening on API servers
All on CommonsWiki
Averaging about 1 per hour
Stack trace is not very useful—may have to grep mwlog to get more info—not sure who to tag :(
Log message normalization would be nice to have, but still easy to filter

Details

Request URL: https://commons.wikimedia.org/w/api.php

Related Objects

Mentioned Here: T283045: LocalFile::lock() rearchitecting

Event Timeline

thcipriani created this task.Nov 4 2021, 3:36 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 4 2021, 3:36 PM

thcipriani moved this task from Untriaged to Nov 2021 on the Wikimedia-production-error board.Nov 18 2021, 4:53 PM

Without further parameter on the request url it is hard to tell which api module is affected, but this is commons and long transactions on upload was improved with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/735762 (part of T283045)

Would suggest to treat is as fixed, but would be nice if the log does not show this error message on commons any longer

Umherirrender merged a task: T282173: Possible uptick in "DBTransactionSizeError: Transaction spent [n] second(s) in writes, exceeding the limit of 3".Aug 7 2022, 6:19 PM

Umherirrender added subscribers: brennen, BPirkle, Xover and 2 others.

Umherirrender merged a task: T314057: Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent {time}s in writes, exceeding the 3s limit.Aug 7 2022, 7:12 PM

As per last comment assume this was a sympton of the general issue with the network/swift backend and database transaction

Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent 14.199902057648 second(s) in writes, exceeding the limit of 3Closed, ResolvedPublicPRODUCTION ERRORActions