Page MenuHomePhabricator
Create Task
Maniphest T349947

Deadlocks in LinkTargetStore::acquireLinkTargetId since 2023-10-18
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
labels.normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction
Function: MediaWiki\Linker\LinkTargetStore::acquireLinkTargetId
Query: Query: INSERT IGNORE INTO `linktarget` (lt_namespace,lt_title) VALUES (0,'X')
error.stack_trace
from /srv/mediawiki/php-1.42.0-wmf.2/includes/libs/rdbms/database/Database.php(1236)
#0 /srv/mediawiki/php-1.42.0-wmf.2/includes/libs/rdbms/database/Database.php(1220): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.42.0-wmf.2/includes/libs/rdbms/database/Database.php(1194): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.42.0-wmf.2/includes/libs/rdbms/database/Database.php(679): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.42.0-wmf.2/includes/libs/rdbms/database/Database.php(1519): Wikimedia\Rdbms\Database->query(Wikimedia\Rdbms\Query, string)
#4 /srv/mediawiki/php-1.42.0-wmf.2/includes/libs/rdbms/database/DBConnRef.php(119): Wikimedia\Rdbms\Database->insert(string, array, string, array)
#5 /srv/mediawiki/php-1.42.0-wmf.2/includes/libs/rdbms/database/DBConnRef.php(407): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.42.0-wmf.2/includes/libs/rdbms/querybuilder/InsertQueryBuilder.php(345): Wikimedia\Rdbms\DBConnRef->insert(string, array, string, array)
#7 /srv/mediawiki/php-1.42.0-wmf.2/includes/linker/LinkTargetStore.php(164): Wikimedia\Rdbms\InsertQueryBuilder->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.2/includes/deferred/LinksUpdate/GenericPageLinksTable.php(145): MediaWiki\Linker\LinkTargetStore->acquireLinkTargetId(MediaWiki\Title\Title, Wikimedia\Rdbms\DBConnRef)
#9 /srv/mediawiki/php-1.42.0-wmf.2/includes/deferred/LinksUpdate/LinksTable.php(380): MediaWiki\Deferred\LinksUpdate\GenericPageLinksTable->insertLink(array)
#10 /srv/mediawiki/php-1.42.0-wmf.2/includes/deferred/LinksUpdate/LinksUpdate.php(223): MediaWiki\Deferred\LinksUpdate\LinksTable->update()
#11 /srv/mediawiki/php-1.42.0-wmf.2/includes/deferred/LinksUpdate/LinksUpdate.php(183): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doIncrementalUpdate()
#12 /srv/mediawiki/php-1.42.0-wmf.2/includes/deferred/DeferredUpdates.php(463): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doUpdate()
#13 /srv/mediawiki/php-1.42.0-wmf.2/includes/deferred/RefreshSecondaryDataUpdate.php(103): DeferredUpdates::attemptUpdate(MediaWiki\Deferred\LinksUpdate\LinksUpdate)
#14 /srv/mediawiki/php-1.42.0-wmf.2/includes/deferred/DeferredUpdates.php(463): RefreshSecondaryDataUpdate->doUpdate()
#15 /srv/mediawiki/php-1.42.0-wmf.2/includes/Storage/DerivedPageDataUpdater.php(1837): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate)
#16 /srv/mediawiki/php-1.42.0-wmf.2/includes/page/WikiPage.php(2082): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(array)
#17 /srv/mediawiki/php-1.42.0-wmf.2/includes/jobqueue/jobs/RefreshLinksJob.php(244): WikiPage->doSecondaryDataUpdates(array)
#18 /srv/mediawiki/php-1.42.0-wmf.2/includes/jobqueue/jobs/RefreshLinksJob.php(153): RefreshLinksJob->runForTitle(MediaWiki\Title\Title)
#19 /srv/mediawiki/php-1.42.0-wmf.2/extensions/EventBus/includes/JobExecutor.php(83): RefreshLinksJob->run()
#20 /srv/mediawiki/rpc/RunSingleJob.php(77): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#21 {main}
Impact

Happened before sporadically but became more frequent around 2023-10-18. (114 times in the last 10 days so still not anything tragic.) Happening during links update, so probably the job just gets retried -> no user impact.

Related Objects

Event Timeline

Tgr created this task.Oct 28 2023, 2:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2023, 2:33 AM
Umherirrender added subscribers: Ladsgroup, Umherirrender.Oct 28 2023, 4:00 PM

Does that happens to specific wikis? On that day T345732: Turn on write both for beta and production was changed to include more wikis, that could bring more load and deadlocks.

Ladsgroup added a comment.Oct 30 2023, 9:52 PM

Yeah, we have turned on write both for many wikis and since it's empty, there is a high chance of concurrency issues and stuff like that. It'll cool down once maint scripts migrate everything.

Umherirrender added a comment.Oct 31 2023, 7:37 PM

I would not assume the table is empty as the templatelinks table is already using the linktarget table, but templates are not often used as links, so the insert rate could be high.
Maybe it is better to have the the maintenance script already done some kind of the table before start writting new, but that would need another run to fill possible gaps when the script finished, so that for other reasons not a good idea.

So this is some kind of "stalled" and waiting for T345733: Run MigrateLinksTable for pagelinks in beta and production to get another look in the logs if everything works fine.

Ladsgroup added a comment.Oct 31 2023, 7:46 PM

I mean pagelinks columns being empty and given that links have a completely different usage pattern than templates, the insertion rate of linktarget will be high regardless. But yeah, let's just wait until the script is finished.

Umherirrender unsubscribed.Nov 29 2023, 10:19 PM
Krinkle added a project: MediaWiki-Engineering.Feb 8 2024, 4:10 PM
Krinkle moved this task from Untriaged to October 2023 on the Wikimedia-production-error board.
Krinkle subscribed.

"Revision backend" is currently listed as unowned on mw:Maintainers. It was previously maintained by Core Platform Team with Daniel actively leading refactoring and code review there.

MSantos moved this task from Inbox, needs triage to Needs Input (waiting) on the MediaWiki-Engineering board.Feb 14 2024, 2:34 PM
daniel subscribed.EditedMar 27 2024, 1:57 PM
In T349947#9525518, @Krinkle wrote:

"Revision backend" is currently listed as unowned on mw:Maintainers. It was previously maintained by Core Platform Team with Daniel actively leading refactoring and code review there.

I did a lot of work on RevisionStore/PageSture/LinkTarget, but the linktarget table and LinkTargetStore class came later, as part of the the link table normalization work that Amir did. I don't know who should own it, I'm just saying that I don't know much about it off-hand.

Umherirrender closed this task as Resolved.Jun 20 2024, 7:41 PM

Migration has finished, table should not throw deadlocks anymore. If there are deadlocks that is a real issue now.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits