Database deadlock in Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer::insertNonExistingRecordsIntoMaster query
Open, LowPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	thcipriani
	May 19 2021, 6:53 PM

Description

Error

mwversion: 1.37.0-wmf.6
reqId: 906b2c11-a61e-41ea-8d30-7355c63ff5cc
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1104)
Function: Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer::insertNonExistingRecordsInto

exception.trace

from /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/database/Database.php(1759)
#0 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/database/Database.php(1743): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/database/Database.php(1718): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/database/Database.php(1282): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/database/Database.php(2423): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/database/Database.php(2383): Wikimedia\Rdbms\Database->doInsertNonConflicting(string, array, string)
#5 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/Sql/Terms/Util/ReplicaMasterAwareRecordIdsAcquirer.php(318): Wikimedia\Rdbms\Database->insert(string, array, string, array)
#6 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/Sql/Terms/Util/ReplicaMasterAwareRecordIdsAcquirer.php(181): Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer->insertNonExistingRecordsIntoMaster(array)
#7 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/Sql/Terms/Util/ReplicaMasterAwareRecordIdsAcquirer.php(131): Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer->insertNonExistingRecords(array, NULL)
#8 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermInLangIdsAcquirer.php(253): Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer->acquireIds(array)
#9 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermInLangIdsAcquirer.php(225): Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsAcquirer->acquireTextInLangIds(array, Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer)
#10 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermInLangIdsAcquirer.php(417): Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsAcquirer->mapToTextInLangIds(array, Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer)
#11 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermInLangIdsAcquirer.php(62): Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsAcquirer->mapTermsArrayToTermIds(array)
#12 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermStoreWriterBase.php(155): Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsAcquirer->acquireTermInLangIds(array, Closure)
#13 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermStoreWriterBase.php(62): Wikibase\Lib\Store\Sql\Terms\DatabaseTermStoreWriterBase->acquireAndInsertTerms(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Term\Fingerprint)
#14 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseItemTermStoreWriter.php(23): Wikibase\Lib\Store\Sql\Terms\DatabaseTermStoreWriterBase->store(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Term\Fingerprint)
#15 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/lib/includes/Store/ItemTermStoreWriterAdapter.php(32): Wikibase\Lib\Store\Sql\Terms\DatabaseItemTermStoreWriter->storeTerms(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Term\Fingerprint)
#16 /srv/mediawiki/php-1.37.0-wmf.6/extensions/Wikibase/repo/includes/Content/DataUpdateAdapter.php(60): Wikibase\Lib\Store\ItemTermStoreWriterAdapter->saveTermsOfEntity(Wikibase\DataModel\Entity\Item)
#17 /srv/mediawiki/php-1.37.0-wmf.6/includes/deferred/DeferredUpdates.php(513): Wikibase\Repo\Content\DataUpdateAdapter->doUpdate()
#18 /srv/mediawiki/php-1.37.0-wmf.6/includes/deferred/RefreshSecondaryDataUpdate.php(101): DeferredUpdates::attemptUpdate(Wikibase\Repo\Content\DataUpdateAdapter, Wikimedia\Rdbms\LBFactoryMulti)
#19 /srv/mediawiki/php-1.37.0-wmf.6/includes/deferred/DeferredUpdates.php(513): RefreshSecondaryDataUpdate->doUpdate()
#20 /srv/mediawiki/php-1.37.0-wmf.6/includes/deferred/DeferredUpdates.php(390): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#21 /srv/mediawiki/php-1.37.0-wmf.6/includes/deferred/DeferredUpdates.php(221): DeferredUpdates::run(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#22 /srv/mediawiki/php-1.37.0-wmf.6/includes/deferred/DeferredUpdatesScope.php(264): DeferredUpdates::{closure}(RefreshSecondaryDataUpdate, integer)
#23 /srv/mediawiki/php-1.37.0-wmf.6/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#24 /srv/mediawiki/php-1.37.0-wmf.6/includes/deferred/DeferredUpdates.php(242): DeferredUpdatesScope->processUpdates(integer, Closure)
#25 /srv/mediawiki/php-1.37.0-wmf.6/includes/MediaWiki.php(1128): DeferredUpdates::doUpdates(string)
#26 /srv/mediawiki/php-1.37.0-wmf.6/includes/MediaWiki.php(838): MediaWiki->restInPeace()
#27 /srv/mediawiki/php-1.37.0-wmf.6/api.php(125): MediaWiki->doPostOutputShutdown()
#28 /srv/mediawiki/php-1.37.0-wmf.6/api.php(45): wfApiMain()
#29 /srv/mediawiki/w/api.php(3): require(string)
#30 {main}

Impact

Happening at a low volume (35 in a few weeks)
started happening in 1.37.0-wmf.5

Notes

Details

Request URL: https://www.wikidata.org/w/api.php

Related Objects

Mentioned In: T365055: [REPO][SW] DatabaseTermStoreWriterBase::acquireAndInsertTerms() shouldn't lock all rows for the item
T351802: WPP: Make term store database configurable
Blog Post: Production Excellence #32: May 2021
Mentioned Here: T355877: [CLIENT] Investigate potential references to wrong ParserOutput in WikibaseClient code

Event Timeline

thcipriani created this task.May 19 2021, 6:53 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 19 2021, 6:53 PM

Maintenance_bot added a project: Wikidata.May 19 2021, 7:45 PM

Restricted Application added a project: [DEPRECATED] wdwb-tech. · View Herald TranscriptMay 19 2021, 7:45 PM

Lucas_Werkmeister_WMDE subscribed.May 20 2021, 1:58 PM

Addshore subscribed.May 20 2021, 2:05 PM

I think this might not actually be a new error?

It certainly looks new when searching for ReplicaMasterAwareRecordIdsAcquirer::insertNonExistingRecordsInto (all screenshots are since 2021-04-20 00:00:00, i.e. covering some 30 days):

But if we search for ReplicaMasterAwareRecordIdsAcquirer instead the picture looks rather different:

And in fact the method name is insertNonExistingRecordsIntoMaster (todo rename, voice and tone), and if we search for ReplicaMasterAwareRecordIdsAcquirer::insertNonExistingRecordsIntoMaster:

I’m not sure why the few insertNonExistingRecordsInto don’t seem to show up in that last screenshot – but anyways, deadlocks while acquiring terms in the Wikibase term store are not a new issue as far as I can tell, something has just caused the method name to sometimes be truncated (dropping the last word) starting with wmf.5.

Though I can’t seem to find any open task for these deadlocks in the term store, so this isn’t a duplicate issue either.

Addshore moved this task from Inbox to Triaged Low (0-50) on the [DEPRECATED] wdwb-tech board.May 25 2021, 11:13 AM

Addshore removed a project: Wikidata-Campsite.May 25 2021, 11:16 AM

thcipriani moved this task from Untriaged to May 2021 on the Wikimedia-production-error board.Jun 9 2021, 6:45 PM

Krinkle mentioned this in Blog Post: Production Excellence #32: May 2021.Jun 21 2021, 1:31 AM

Michael subscribed.Jun 29 2021, 11:46 AM

Addshore triaged this task as Low priority.Jul 14 2021, 11:12 AM

Addshore moved this task from Triaged Low (0-50) to Triaged Medium (50+) on the [DEPRECATED] wdwb-tech board.Aug 3 2021, 1:35 PM

Krinkle renamed this task from Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1104) Function: Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer::insertNonExistingRecordsIntoMaster Query: INSERT IGNORE INTO `wbt_text_in_lang` (wbxl_text_id,wbxl_language) VALUES ('####'y,'es'),('####x','fr') to Database deadlock in Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer::insertNonExistingRecordsIntoMaster query.Oct 27 2021, 6:36 PM

Addshore unsubscribed.Jun 27 2023, 12:43 PM

Krinkle moved this task from May 2021 to Older on the Wikimedia-production-error board.Aug 17 2023, 3:49 PM

Lucas_Werkmeister_WMDE mentioned this in T351802: WPP: Make term store database configurable.Nov 22 2023, 2:56 PM

As the data stored in the database are from parsing a page (via SecondaryDataUpdates), the fixes for T355877: [CLIENT] Investigate potential references to wrong ParserOutput in WikibaseClient code could be fix this as well.

Umherirrender unsubscribed.Apr 4 2024, 6:30 PM

Lucas_Werkmeister_WMDE mentioned this in T365055: [REPO][SW] DatabaseTermStoreWriterBase::acquireAndInsertTerms() shouldn't lock all rows for the item.May 16 2024, 12:51 PM

	F34461793: Screenshot_2021-05-20 mediawiki-errors - Elastic.png
	May 20 2021, 3:26 PM

	F34461791: Screenshot_2021-05-20 mediawiki-errors - Elastic.png
	May 20 2021, 3:26 PM

	F34461798: Screenshot_2021-05-20 mediawiki-errors - Elastic.png
	May 20 2021, 3:26 PM

Database deadlock in Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer::insertNonExistingRecordsIntoMaster queryOpen, LowPublicPRODUCTION ERRORActions