Fatal "DBQueryError: Lock wait timeout exceeded" from WikiPage::updateCategoryCounts after some edits
Closed, DuplicatePublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Zabe
	May 7 2024, 4:30 PM

Description

Error

service.version: 1.43.0-wmf.3
trace.id: 2a45caa7-580b-41ce-a521-af4af278e3a4
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent {time}s in writes, exceeding the 3s limit

error.stack_trace

from /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1400)
#0 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/lbfactory/LBFactory.php(325): Wikimedia\Rdbms\LoadBalancer->approvePrimaryChanges(integer, string)
#1 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(297): Wikimedia\Rdbms\LBFactory->commitPrimaryChanges(string, integer)
#2 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(188): MediaWiki\MediaWikiEntryPoint->commitMainTransaction()
#3 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(171): MediaWiki\MediaWikiEntryPoint->doPrepareForOutput()
#4 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWiki.php(90): MediaWiki\MediaWikiEntryPoint->prepareForOutput()
#5 /srv/mediawiki/php-1.43.0-wmf.3/includes/api/ApiMain.php(952): MediaWiki::preOutputCommit(MediaWiki\Context\DerivativeContext)
#6 /srv/mediawiki/php-1.43.0-wmf.3/includes/api/ApiMain.php(895): ApiMain->executeActionWithErrorHandling()
#7 /srv/mediawiki/php-1.43.0-wmf.3/includes/api/ApiEntryPoint.php(158): ApiMain->execute()
#8 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(199): MediaWiki\Api\ApiEntryPoint->execute()
#9 /srv/mediawiki/php-1.43.0-wmf.3/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#10 /srv/mediawiki/w/api.php(3): require(string)
#11 {main}

Impact

Notes

Prevously in 2022: T300303: Deadlock in WikiPage::updateCategoryCounts query

Details

Request URL: https://commons.wikimedia.org/w/api.php

Related Objects

Mentioned In: T400908: Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transactionFunction: MediaWiki\Page\WikiPage::updateCategoryCountsQuery
T395745: MediaWiki periodic job recount-categories failed
T221795: Refactor Category::refreshCounts logic to a job and simplify
T365303: Move update of category members count to CategoryMembershipChangeJob
T311308: DBTransactionSizeError: Transaction spent {time}s in writes, exceeding the 3s limit (via Action API globalpreferences module)
T364360: Intermittent timeouts when saving at Commons
Mentioned Here: T365303: Move update of category members count to CategoryMembershipChangeJob
T311308: DBTransactionSizeError: Transaction spent {time}s in writes, exceeding the 3s limit (via Action API globalpreferences module)
T300303: Deadlock in WikiPage::updateCategoryCounts query

Event Timeline

Zabe created this task.May 7 2024, 4:30 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2024, 4:30 PM

Also

Error

mwversion: 1.43.0-wmf.3
reqId: 46752d31-9b92-4362-a2eb-b0812d488458
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1205: Lock wait timeout exceeded; try restarting transaction
Function: WikiPage::updateCategoryCounts
Query: UPDATE  `category` SET cat_pages = cat_pages + 1,cat_files = cat_files + 1 WHERE c

exception.trace

from /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/Database.php(1203)
#0 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/Database.php(1187): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/Database.php(1161): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/Database.php(652): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/Database.php(1501): Wikimedia\Rdbms\Database->query(Wikimedia\Rdbms\Query, string)
#4 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/DBConnRef.php(126): Wikimedia\Rdbms\Database->update(string, array, array, string, array)
#5 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/database/DBConnRef.php(420): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.43.0-wmf.3/includes/libs/rdbms/querybuilder/UpdateQueryBuilder.php(332): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string, array)
#7 /srv/mediawiki/php-1.43.0-wmf.3/includes/page/WikiPage.php(2969): Wikimedia\Rdbms\UpdateQueryBuilder->execute()
#8 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/LinksUpdate/CategoryLinksTable.php(320): WikiPage->updateCategoryCounts(array, array, integer)
#9 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/LinksUpdate/CategoryLinksTable.php(287): MediaWiki\Deferred\LinksUpdate\CategoryLinksTable->updateCategoryCounts(array, array)
#10 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/LinksUpdate/LinksTable.php(392): MediaWiki\Deferred\LinksUpdate\CategoryLinksTable->finishUpdate()
#11 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/LinksUpdate/LinksUpdate.php(225): MediaWiki\Deferred\LinksUpdate\LinksTable->update()
#12 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/LinksUpdate/LinksUpdate.php(185): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doIncrementalUpdate()
#13 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdates.php(486): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doUpdate()
#14 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/RefreshSecondaryDataUpdate.php(110): MediaWiki\Deferred\DeferredUpdates::attemptUpdate(MediaWiki\Deferred\LinksUpdate\LinksUpdate)
#15 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdates.php(486): RefreshSecondaryDataUpdate->doUpdate()
#16 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdates.php(198): MediaWiki\Deferred\DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate)
#17 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdates.php(285): MediaWiki\Deferred\DeferredUpdates::run(RefreshSecondaryDataUpdate)
#18 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdatesScope.php(266): MediaWiki\Deferred\DeferredUpdates::MediaWiki\Deferred\{closure}(RefreshSecondaryDataUpdate, integer)
#19 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdatesScope.php(198): MediaWiki\Deferred\DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#20 /srv/mediawiki/php-1.43.0-wmf.3/includes/deferred/DeferredUpdates.php(304): MediaWiki\Deferred\DeferredUpdatesScope->processUpdates(integer, Closure)
#21 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(672): MediaWiki\Deferred\DeferredUpdates::doUpdates()
#22 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(495): MediaWiki\MediaWikiEntryPoint->restInPeace()
#23 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(453): MediaWiki\MediaWikiEntryPoint->doPostOutputShutdown()
#24 /srv/mediawiki/php-1.43.0-wmf.3/includes/MediaWikiEntryPoint.php(208): MediaWiki\MediaWikiEntryPoint->postOutputShutdown()
#25 /srv/mediawiki/php-1.43.0-wmf.3/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#26 /srv/mediawiki/w/api.php(3): require(string)
#27 {main}

There were like 10000 errors of these kinds within the last hour.

hnowlan subscribed.May 7 2024, 4:51 PM

andrea.denisse subscribed.May 7 2024, 4:52 PM

This is spiking latency, 5xx and worker saturation in eqiad api servers

Aklapper mentioned this in T364360: Intermittent timeouts when saving at Commons.May 7 2024, 5:22 PM

Looks like it started on Sunday when there were no deploys:

2024-05-07-3s-limit.png (635×927 px, 46 KB)

Last refactor was in T300303: Deadlock in WikiPage::updateCategoryCounts query

It seems that one category is added to many pages/files. This could be a new gadget or tool.
It is possible to group the Api-UserAgent to see if that is related? Maybe the api requests should not done in parallel, if that is related.

This looks similar to T311308

Zabe merged a task: T364360: Intermittent timeouts when saving at Commons.May 9 2024, 6:57 PM

Zabe added a subscriber: Nurg.

andrea.denisse unsubscribed.May 13 2024, 5:22 PM

Lowering priority since it seems like this error is no longer happening at the same level—were there any changes related to this task that would have cause that? Or is this driven by traffic?

thcipriani moved this task from Untriaged to May 2024 on the Wikimedia-production-error board.May 16 2024, 3:29 PM

Umherirrender mentioned this in T365303: Move update of category members count to CategoryMembershipChangeJob.May 17 2024, 10:34 PM

A possible solution to this problem is this: T365303: Move update of category members count to CategoryMembershipChangeJob

Umherirrender unsubscribed.Jun 16 2024, 11:33 AM

Krinkle renamed this task from Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent {time}s in writes, exceeding the 3s limit to Fatal "DBQueryError: Lock wait timeout exceeded" from WikiPage::updateCategoryCounts after some edits.Mar 22 2025, 9:08 AM

Krinkle updated the task description. (Show Details)

Krinkle mentioned this in T221795: Refactor Category::refreshCounts logic to a job and simplify.Mar 22 2025, 9:28 AM

A_smart_kitten mentioned this in T395745: MediaWiki periodic job recount-categories failed.Jun 2 2025, 11:36 AM

Umherirrender mentioned this in T400908: Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transactionFunction: MediaWiki\Page\WikiPage::updateCategoryCountsQuery.Jul 31 2025, 6:00 PM

Umherirrender merged a task: T400908: Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transactionFunction: MediaWiki\Page\WikiPage::updateCategoryCountsQuery.Aug 25 2025, 7:32 PM

Umherirrender closed this task as a duplicate of T365303: Move update of category members count to CategoryMembershipChangeJob.

Umherirrender added subscribers: brennen, Mbch331, Count_Count, daniel.