Page MenuHomePhabricator
Create Task
Maniphest T300303

Deadlock in WikiPage::updateCategoryCounts query
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1138)
Function: WikiPage::updateCategoryCounts
Query: UPDATE  `category` SET cat_pages = cat_pages - 1,cat_files =
exception.trace
from /srv/mediawiki/php-1.38.0-wmf.19/includes/libs/rdbms/database/Database.php(1810)
#0 /srv/mediawiki/php-1.38.0-wmf.19/includes/libs/rdbms/database/Database.php(1794): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.38.0-wmf.19/includes/libs/rdbms/database/Database.php(1768): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.38.0-wmf.19/includes/libs/rdbms/database/Database.php(1305): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.38.0-wmf.19/includes/libs/rdbms/database/Database.php(2670): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.38.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(69): Wikimedia\Rdbms\Database->update(string, array, string, string)
#5 /srv/mediawiki/php-1.38.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(381): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.38.0-wmf.19/includes/page/WikiPage.php(3143): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string)
#7 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/LinksUpdate/CategoryLinksTable.php(296): WikiPage->updateCategoryCounts(array, array, integer)
#8 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/LinksUpdate/CategoryLinksTable.php(264): MediaWiki\Deferred\LinksUpdate\CategoryLinksTable->updateCategoryCounts()
#9 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/LinksUpdate/LinksTable.php(391): MediaWiki\Deferred\LinksUpdate\CategoryLinksTable->finishUpdate()
#10 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/LinksUpdate/LinksUpdate.php(292): MediaWiki\Deferred\LinksUpdate\LinksTable->update()
#11 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/LinksUpdate/LinksUpdate.php(252): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doIncrementalUpdate()
#12 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/DeferredUpdates.php(529): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doUpdate()
#13 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/RefreshSecondaryDataUpdate.php(103): DeferredUpdates::attemptUpdate(MediaWiki\Deferred\LinksUpdate\LinksUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#14 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/DeferredUpdates.php(529): RefreshSecondaryDataUpdate->doUpdate()
#15 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/DeferredUpdates.php(414): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#16 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/DeferredUpdates.php(227): DeferredUpdates::run(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#17 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/DeferredUpdatesScope.php(264): DeferredUpdates::{closure}(RefreshSecondaryDataUpdate, integer)
#18 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#19 /srv/mediawiki/php-1.38.0-wmf.19/includes/deferred/DeferredUpdates.php(248): DeferredUpdatesScope->processUpdates(integer, Closure)
#20 /srv/mediawiki/php-1.38.0-wmf.19/includes/MediaWiki.php(1114): DeferredUpdates::doUpdates()
#21 /srv/mediawiki/php-1.38.0-wmf.19/includes/MediaWiki.php(839): MediaWiki->restInPeace()
#22 /srv/mediawiki/php-1.38.0-wmf.19/includes/MediaWiki.php(588): MediaWiki->doPostOutputShutdown()
#23 /srv/mediawiki/php-1.38.0-wmf.19/index.php(53): MediaWiki->run()
#24 /srv/mediawiki/php-1.38.0-wmf.19/index.php(46): wfIndexMain()
#25 /srv/mediawiki/w/index.php(3): require(string)
#26 {main}
Impact

Unclear.

Notes

Noticing a handful of these in 1.38.0-wmf.19 (T293960).

Related to T281714?

Details

Request URL
https://commons.wikimedia.org/w/index.php?title=*&action=submit
SubjectRepoBranchLines +/-
WikiPage: Simplify added/deleted logic in ::updateCategoryCounts()mediawiki/coremaster+33 -35
WikiPage: Use primary key for updates in ::updateCategoryCountsmediawiki/coremaster+14 -9
Customize query in gerrit

Related Objects

Event Timeline

brennen created this task.Jan 27 2022, 8:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 27 2022, 8:36 PM
brennen updated the task description. (Show Details)Jan 27 2022, 8:37 PM
Zabe subscribed.Jan 27 2022, 9:09 PM
Umherirrender subscribed.Jan 27 2022, 9:11 PM

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/754065 would reduce the (concurrent) updates on page moves.

But according to the request url this is from an edit which removes a category from a file page, not from move. So this could be unrelated to the situation.

The impact could be, that one category counter is wrong, which could itself fixing within a month when T299823: Regularly run recountCategories.php on Wikimedia wikis via systemd timer is installed.

brennen moved this task from Backlog to Logs/Train on the User-brennen board.Jan 31 2022, 5:53 PM
Krinkle moved this task from Untriaged to Jan 2022 on the Wikimedia-production-error board.Feb 4 2022, 3:10 AM
Lucas_Werkmeister_WMDE mentioned this in T301434: Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1138)Function: WikiPage::updateCategoryCountsQuery: UPDATE `category` SET cat_pages = cat_pages + 1,cat_files = cat_files + 1 WHERE cat_title IN ('Images_from_Deskthority','PD-author','Released_into_the_public_domain_by_the_copyright_holder_missing_SDC_copyright_license') .Feb 10 2022, 3:11 PM
Krinkle merged a task: T301434: Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1138)Function: WikiPage::updateCategoryCountsQuery: UPDATE `category` SET cat_pages = cat_pages + 1,cat_files = cat_files + 1 WHERE cat_title IN ('Images_from_Deskthority','PD-author','Released_into_the_public_domain_by_the_copyright_holder_missing_SDC_copyright_license') .Apr 14 2022, 3:35 PM
Krinkle added subscribers: jeena, tstarling, Lucas_Werkmeister_WMDE.
hashar subscribed.May 17 2022, 7:38 AM

One occurrence today with 1.39.0-wmf.10, there are two messages shown for the same reqId:

reqId: 2719cab7-1df6-4c0b-9434-7b78f04fdea3
Find reqId in Logstash

normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1160)
Function: WikiPage::updateCategoryCounts
Query: UPDATE  `category` SET cat_pages = cat_pages - 1,cat_subcats
exception.trace
from /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1743)
#0 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1727): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1701): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1202): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(2603): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update(string, array, string, string)
#5 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(341): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/WikiPage.php(3165): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string)
#7 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/LinksUpdate/CategoryLinksTable.php(309): WikiPage->updateCategoryCounts(array, array, integer)
#8 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/LinksUpdate/CategoryLinksTable.php(275): MediaWiki\Deferred\LinksUpdate\CategoryLinksTable->updateCategoryCounts(array, array)
#9 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/LinksUpdate/LinksTable.php(398): MediaWiki\Deferred\LinksUpdate\CategoryLinksTable->finishUpdate()
#10 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/LinksUpdate/LinksUpdate.php(294): MediaWiki\Deferred\LinksUpdate\LinksTable->update()
#11 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/LinksUpdate/LinksUpdate.php(254): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doIncrementalUpdate()
#12 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/DeferredUpdates.php(536): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doUpdate()
#13 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/RefreshSecondaryDataUpdate.php(103): DeferredUpdates::attemptUpdate(MediaWiki\Deferred\LinksUpdate\LinksUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#14 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/DeferredUpdates.php(536): RefreshSecondaryDataUpdate->doUpdate()
#15 /srv/mediawiki/php-1.39.0-wmf.10/includes/Storage/DerivedPageDataUpdater.php(1783): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#16 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/WikiPage.php(2207): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(array)
#17 /srv/mediawiki/php-1.39.0-wmf.10/includes/jobqueue/jobs/RefreshLinksJob.php(242): WikiPage->doSecondaryDataUpdates(array)
#18 /srv/mediawiki/php-1.39.0-wmf.10/includes/jobqueue/jobs/RefreshLinksJob.php(163): RefreshLinksJob->runForTitle(Title)
#19 /srv/mediawiki/php-1.39.0-wmf.10/extensions/EventBus/includes/JobExecutor.php(79): RefreshLinksJob->run()
#20 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#21 {main}

And

normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1160)
Function: WikiPage::updateCategoryCounts
Query: UPDATE  `category` SET cat_pages = cat_pages - 1,cat_subcats
exception.trace
from /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1743)
#0 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1727): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1701): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1202): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(2603): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update(string, array, string, string)
#5 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(341): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/WikiPage.php(3165): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string)
#7 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/LinksUpdate/CategoryLinksTable.php(309): WikiPage->updateCategoryCounts(array, array, integer)
#8 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/LinksUpdate/CategoryLinksTable.php(275): MediaWiki\Deferred\LinksUpdate\CategoryLinksTable->updateCategoryCounts(array, array)
#9 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/LinksUpdate/LinksTable.php(398): MediaWiki\Deferred\LinksUpdate\CategoryLinksTable->finishUpdate()
#10 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/LinksUpdate/LinksUpdate.php(294): MediaWiki\Deferred\LinksUpdate\LinksTable->update()
#11 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/LinksUpdate/LinksUpdate.php(254): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doIncrementalUpdate()
#12 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/DeferredUpdates.php(536): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doUpdate()
#13 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/RefreshSecondaryDataUpdate.php(103): DeferredUpdates::attemptUpdate(MediaWiki\Deferred\LinksUpdate\LinksUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#14 /srv/mediawiki/php-1.39.0-wmf.10/includes/deferred/DeferredUpdates.php(536): RefreshSecondaryDataUpdate->doUpdate()
#15 /srv/mediawiki/php-1.39.0-wmf.10/includes/Storage/DerivedPageDataUpdater.php(1783): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#16 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/WikiPage.php(2207): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(array)
#17 /srv/mediawiki/php-1.39.0-wmf.10/includes/jobqueue/jobs/RefreshLinksJob.php(242): WikiPage->doSecondaryDataUpdates(array)
#18 /srv/mediawiki/php-1.39.0-wmf.10/includes/jobqueue/jobs/RefreshLinksJob.php(163): RefreshLinksJob->runForTitle(Title)
#19 /srv/mediawiki/php-1.39.0-wmf.10/extensions/EventBus/includes/JobExecutor.php(79): RefreshLinksJob->run()
#20 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#21 {main}

I am guessing due to a retry?

Umherirrender unsubscribed.Jun 19 2022, 7:37 PM
Ladsgroup mentioned this in T306636: DBQueryError: Error 1213: Deadlock found in UserOptionsManager::saveOptionsInternal.Aug 11 2022, 9:22 AM
gerritbot added a comment.Dec 6 2022, 8:02 PM

Change 865164 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] WikiPage: Use primary key for updates in ::updateCategoryCounts

https://gerrit.wikimedia.org/r/865164

gerritbot added a project: Patch-For-Review.Dec 6 2022, 8:02 PM
Umherirrender added a comment.Dec 6 2022, 8:32 PM
In T300303#8448445, @gerritbot wrote:

Change 865164 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] WikiPage: Use primary key for updates in ::updateCategoryCounts

https://gerrit.wikimedia.org/r/865164

This is just an idea to get an effect without many lines to change.
If that does not help or is not wanted the next step would/could be to use a mergable defered update similiar to how the edit count is handled (UserEditCountUpdate::class), which could collect some needed updates post send and commit all together.

Krinkle renamed this task from DBQueryError: Deadlock found when trying to get lock; try restarting transaction (db1138)Function: WikiPage::updateCategoryCountsQuery: UPDATE `category` SET cat_pages = cat_pages - 1,cat_files = cat_files - 1 WHERE cat_title = '[title]' to Deadlock in WikiPage::updateCategoryCounts query.Feb 16 2023, 10:14 PM
gerritbot added a comment.Feb 16 2023, 10:21 PM

Change 889883 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] WikiPage: Simplify added/deleted logic in ::updateCategoryCounts()

https://gerrit.wikimedia.org/r/889883

gerritbot added a comment.Feb 16 2023, 10:34 PM

Change 865164 merged by jenkins-bot:

[mediawiki/core@master] WikiPage: Use primary key for updates in ::updateCategoryCounts

https://gerrit.wikimedia.org/r/865164

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.24; 2023-02-20).Feb 16 2023, 11:00 PM
gerritbot added a comment.Feb 17 2023, 7:59 PM

Change 889883 merged by jenkins-bot:

[mediawiki/core@master] WikiPage: Simplify added/deleted logic in ::updateCategoryCounts()

https://gerrit.wikimedia.org/r/889883

Umherirrender closed this task as Resolved.Feb 17 2023, 8:09 PM
Umherirrender claimed this task.
Maintenance_bot removed a project: Patch-For-Review.Feb 17 2023, 8:10 PM
Umherirrender mentioned this in T364404: Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent {time}s in writes, exceeding the 3s limit.May 7 2024, 7:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits