Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent {time}s in writes, exceeding the 3s limit
Open, HighPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Zabe
	May 7 2024, 4:30 PM

Description

Error

service.version: 1.43.0-wmf.3
trace.id: 2a45caa7-580b-41ce-a521-af4af278e3a4
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent {time}s in writes, exceeding the 3s limit

error.stack_trace

from /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1400)
#0 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/lbfactory/LBFactory.php(325): Wikimedia\Rdbms\LoadBalancer->approvePrimaryChanges(integer, string)
#1 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(297): Wikimedia\Rdbms\LBFactory->commitPrimaryChanges(string, integer)
#2 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(188): MediaWiki\MediaWikiEntryPoint->commitMainTransaction()
#3 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(171): MediaWiki\MediaWikiEntryPoint->doPrepareForOutput()
#4 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWiki.php(90): MediaWiki\MediaWikiEntryPoint->prepareForOutput()
#5 /srv/mediawiki/php-1.43.0-wmf.3/includes/api/ApiMain.php(952): MediaWiki::preOutputCommit(MediaWiki\Context\DerivativeContext)
#6 /srv/mediawiki/php-1.43.0-wmf.3/includes/api/ApiMain.php(895): ApiMain->executeActionWithErrorHandling()
#7 /srv/mediawiki/php-1.43.0-wmf.3/includes/api/ApiEntryPoint.php(158): ApiMain->execute()
#8 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(199): MediaWiki\Api\ApiEntryPoint->execute()
#9 /srv/mediawiki/php-1.43.0-wmf.3/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#10 /srv/mediawiki/w/api.php(3): require(string)
#11 {main}

Impact

Notes

Details

Request URL: https://commons.wikimedia.org/w/api.php

Related Objects

Mentioned In: T365303: Move update of category members count to CategoryMembershipChangeJob
T311308: DBTransactionSizeError: Transaction spent {time}s in writes, exceeding the 3s limit (via Action API globalpreferences module)
T364360: Intermittent timeouts when saving at Commons
Mentioned Here: T365303: Move update of category members count to CategoryMembershipChangeJob
T311308: DBTransactionSizeError: Transaction spent {time}s in writes, exceeding the 3s limit (via Action API globalpreferences module)
T300303: Deadlock in WikiPage::updateCategoryCounts query

Event Timeline

Zabe created this task.May 7 2024, 4:30 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2024, 4:30 PM

Also

Error

mwversion: 1.43.0-wmf.3
reqId: 46752d31-9b92-4362-a2eb-b0812d488458
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1205: Lock wait timeout exceeded; try restarting transaction
Function: WikiPage::updateCategoryCounts
Query: UPDATE  `category` SET cat_pages = cat_pages + 1,cat_files = cat_files + 1 WHERE c

exception.trace

from /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/Database.php(1203)
#0 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/Database.php(1187): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/Database.php(1161): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/Database.php(652): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/Database.php(1501): Wikimedia\Rdbms\Database->query(Wikimedia\Rdbms\Query, string)
#4 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/DBConnRef.php(126): Wikimedia\Rdbms\Database->update(string, array, array, string, array)
#5 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/DBConnRef.php(420): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/querybuilder/UpdateQueryBuilder.php(332): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string, array)
#7 /srv/mediawiki/php-1.43.0-wmf.3/includes/page/WikiPage.php(2969): Wikimedia\Rdbms\UpdateQueryBuilder->execute()
#8 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/LinksUpdate/CategoryLinksTable.php(320): WikiPage->updateCategoryCounts(array, array, integer)
#9 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/LinksUpdate/CategoryLinksTable.php(287): MediaWiki\Deferred\LinksUpdate\CategoryLinksTable->updateCategoryCounts(array, array)
#10 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/LinksUpdate/LinksTable.php(392): MediaWiki\Deferred\LinksUpdate\CategoryLinksTable->finishUpdate()
#11 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/LinksUpdate/LinksUpdate.php(225): MediaWiki\Deferred\LinksUpdate\LinksTable->update()
#12 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/LinksUpdate/LinksUpdate.php(185): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doIncrementalUpdate()
#13 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdates.php(486): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doUpdate()
#14 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/RefreshSecondaryDataUpdate.php(110): MediaWiki\Deferred\DeferredUpdates::attemptUpdate(MediaWiki\Deferred\LinksUpdate\LinksUpdate)
#15 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdates.php(486): RefreshSecondaryDataUpdate->doUpdate()
#16 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdates.php(198): MediaWiki\Deferred\DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate)
#17 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdates.php(285): MediaWiki\Deferred\DeferredUpdates::run(RefreshSecondaryDataUpdate)
#18 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdatesScope.php(266): MediaWiki\Deferred\DeferredUpdates::MediaWiki\Deferred\{closure}(RefreshSecondaryDataUpdate, integer)
#19 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdatesScope.php(198): MediaWiki\Deferred\DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#20 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdates.php(304): MediaWiki\Deferred\DeferredUpdatesScope->processUpdates(integer, Closure)
#21 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(672): MediaWiki\Deferred\DeferredUpdates::doUpdates()
#22 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(495): MediaWiki\MediaWikiEntryPoint->restInPeace()
#23 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(453): MediaWiki\MediaWikiEntryPoint->doPostOutputShutdown()
#24 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(208): MediaWiki\MediaWikiEntryPoint->postOutputShutdown()
#25 /srv/mediawiki/php-1.43.0-wmf.3/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#26 /srv/mediawiki/w/api.php(3): require(string)
#27 {main}

There were like 10000 errors of these kinds within the last hour.

hnowlan subscribed.May 7 2024, 4:51 PM

andrea.denisse subscribed.May 7 2024, 4:52 PM

This is spiking latency, 5xx and worker saturation in eqiad api servers

Aklapper mentioned this in T364360: Intermittent timeouts when saving at Commons.May 7 2024, 5:22 PM

Looks like it started on Sunday when there were no deploys:

2024-05-07-3s-limit.png (635×927 px, 46 KB)

Last refactor was in T300303: Deadlock in WikiPage::updateCategoryCounts query

It seems that one category is added to many pages/files. This could be a new gadget or tool.
It is possible to group the Api-UserAgent to see if that is related? Maybe the api requests should not done in parallel, if that is related.

This looks similar to T311308

Zabe merged a task: T364360: Intermittent timeouts when saving at Commons.May 9 2024, 6:57 PM

Zabe added a subscriber: Nurg.

andrea.denisse unsubscribed.May 13 2024, 5:22 PM

Lowering priority since it seems like this error is no longer happening at the same level—were there any changes related to this task that would have cause that? Or is this driven by traffic?

thcipriani moved this task from Untriaged to May 2024 on the Wikimedia-production-error board.May 16 2024, 3:29 PM

Umherirrender mentioned this in T365303: Move update of category members count to CategoryMembershipChangeJob.May 17 2024, 10:34 PM