Page MenuHomePhabricator

Deadlock found in Function: LinksTable::doWritesQuery: DELETE FROM `page_props`
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1157)
Function: MediaWiki\Deferred\LinksUpdate\LinksTable::doWrites
Query: DELETE FROM `page_props` WHERE (pp_page
exception.trace
from /srv/mediawiki/php-1.38.0-wmf.18/includes/libs/rdbms/database/Database.php(1799)
#0 /srv/mediawiki/php-1.38.0-wmf.18/includes/libs/rdbms/database/Database.php(1783): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.38.0-wmf.18/includes/libs/rdbms/database/Database.php(1758): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.38.0-wmf.18/includes/libs/rdbms/database/Database.php(1302): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.38.0-wmf.18/includes/libs/rdbms/database/Database.php(3746): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/LinksUpdate/LinksTable.php(463): Wikimedia\Rdbms\Database->delete(string, string, string)
#5 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/LinksUpdate/LinksTable.php(390): MediaWiki\Deferred\LinksUpdate\LinksTable->doWrites()
#6 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/LinksUpdate/LinksUpdate.php(292): MediaWiki\Deferred\LinksUpdate\LinksTable->update()
#7 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/LinksUpdate/LinksUpdate.php(252): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doIncrementalUpdate()
#8 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/DeferredUpdates.php(529): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doUpdate()
#9 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/RefreshSecondaryDataUpdate.php(103): DeferredUpdates::attemptUpdate(MediaWiki\Deferred\LinksUpdate\LinksUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#10 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/DeferredUpdates.php(529): RefreshSecondaryDataUpdate->doUpdate()
#11 /srv/mediawiki/php-1.38.0-wmf.18/includes/Storage/DerivedPageDataUpdater.php(1745): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#12 /srv/mediawiki/php-1.38.0-wmf.18/includes/page/WikiPage.php(2191): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(array)
#13 /srv/mediawiki/php-1.38.0-wmf.18/includes/jobqueue/jobs/RefreshLinksJob.php(210): WikiPage->doSecondaryDataUpdates(array)
#14 /srv/mediawiki/php-1.38.0-wmf.18/includes/jobqueue/jobs/RefreshLinksJob.php(129): RefreshLinksJob->runForTitle(Title)
#15 /srv/mediawiki/php-1.38.0-wmf.18/extensions/EventBus/includes/JobExecutor.php(79): RefreshLinksJob->run()
#16 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#17 {main}
Impact
Notes

Based on Logstash/mediawiki-errors query "Deadlock found when trying" AND page_props it seems these are new this/last week. Given there aren't older entries with a different class name than the new MediaWiki\Deferred\LinksUpdate\LinksTable class, this seems most likely fallout from change 743278 ("LinksUpdate refactor", ref T263437).

Details

Request URL
https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle renamed this task from Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1157)Function: MediaWiki\Deferred\LinksUpdate\LinksTable::doWritesQuery: DELETE FROM `page_props` WHERE (pp_page = 8898 AND pp_propname IN ('kartographer_frames','kartographer_links') ) to Deadlock found in Function: MediaWiki\Deferred\LinksUpdate\LinksTable::doWritesQuery: DELETE FROM `page_props`.Jan 20 2022, 4:38 PM
Krinkle renamed this task from Deadlock found in Function: MediaWiki\Deferred\LinksUpdate\LinksTable::doWritesQuery: DELETE FROM `page_props` to Deadlock found in Function: LinksTable::doWritesQuery: DELETE FROM `page_props`.
Krinkle removed a project: Platform Engineering.
Krinkle moved this task from Inbox to Doing: Prio Interrupt on the Performance-Team board.

Searching logstash from Jan 19 to now, I noticed that 668 out of 674 errors come from deleting the properties kartographer_frames and/or kartographer_links.

I was able to reproduce this locally. I imported https://en.wikivoyage.org/wiki/Aegina and its templates, then I edited one of the templates it used, then in the job queue debug log:

2022-01-28T14:01:07.836207+11:00 3464598 [DBQuery] MediaWiki\Deferred\LinksUpdate\LinksTable::doWrites [0.001s] localhost: DELETE FROM `page_props` WHERE (pp_page = 983 AND pp_propname IN ('kartographer_frames','kartographer_links') )

Probably something about the weird data flow in Module:Map confuses refreshLinks.

OK, sorry, it's not a Kartographer issue, it is indeed from the LinksUpdate refactor. New property values are compared to existing values by strict comparison, which doesn't work when an integer is given. The old code used array_diff_assoc() which implicitly converted to string before comparing.

Change 757793 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] Fix page_props DELETE/INSERT on null edit

https://gerrit.wikimedia.org/r/757793

Change 757793 merged by jenkins-bot:

[mediawiki/core@master] Fix page_props DELETE/INSERT on null edit

https://gerrit.wikimedia.org/r/757793

Nikerabbit added a subscriber: Nikerabbit.

We saw this during update.php call during deployment. I'll assume after we deploy the new version, it will have the fix and this issue will not appear again for us.