Page MenuHomePhabricator
Create Task
Maniphest T299662

Deadlock found in Function: LinksTable::doWritesQuery: DELETE FROM `page_props`
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1157)
Function: MediaWiki\Deferred\LinksUpdate\LinksTable::doWrites
Query: DELETE FROM `page_props` WHERE (pp_page
exception.trace
from /srv/mediawiki/php-1.38.0-wmf.18/includes/libs/rdbms/database/Database.php(1799)
#0 /srv/mediawiki/php-1.38.0-wmf.18/includes/libs/rdbms/database/Database.php(1783): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.38.0-wmf.18/includes/libs/rdbms/database/Database.php(1758): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.38.0-wmf.18/includes/libs/rdbms/database/Database.php(1302): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.38.0-wmf.18/includes/libs/rdbms/database/Database.php(3746): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/LinksUpdate/LinksTable.php(463): Wikimedia\Rdbms\Database->delete(string, string, string)
#5 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/LinksUpdate/LinksTable.php(390): MediaWiki\Deferred\LinksUpdate\LinksTable->doWrites()
#6 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/LinksUpdate/LinksUpdate.php(292): MediaWiki\Deferred\LinksUpdate\LinksTable->update()
#7 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/LinksUpdate/LinksUpdate.php(252): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doIncrementalUpdate()
#8 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/DeferredUpdates.php(529): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doUpdate()
#9 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/RefreshSecondaryDataUpdate.php(103): DeferredUpdates::attemptUpdate(MediaWiki\Deferred\LinksUpdate\LinksUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#10 /srv/mediawiki/php-1.38.0-wmf.18/includes/deferred/DeferredUpdates.php(529): RefreshSecondaryDataUpdate->doUpdate()
#11 /srv/mediawiki/php-1.38.0-wmf.18/includes/Storage/DerivedPageDataUpdater.php(1745): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#12 /srv/mediawiki/php-1.38.0-wmf.18/includes/page/WikiPage.php(2191): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(array)
#13 /srv/mediawiki/php-1.38.0-wmf.18/includes/jobqueue/jobs/RefreshLinksJob.php(210): WikiPage->doSecondaryDataUpdates(array)
#14 /srv/mediawiki/php-1.38.0-wmf.18/includes/jobqueue/jobs/RefreshLinksJob.php(129): RefreshLinksJob->runForTitle(Title)
#15 /srv/mediawiki/php-1.38.0-wmf.18/extensions/EventBus/includes/JobExecutor.php(79): RefreshLinksJob->run()
#16 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#17 {main}
Impact
Notes

Based on Logstash/mediawiki-errors query "Deadlock found when trying" AND page_props it seems these are new this/last week. Given there aren't older entries with a different class name than the new MediaWiki\Deferred\LinksUpdate\LinksTable class, this seems most likely fallout from change 743278 ("LinksUpdate refactor", ref T263437).

Details

Request URL
https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php
SubjectRepoBranchLines +/-
Fix page_props DELETE/INSERT on null editmediawiki/coremaster+54 -4
Customize query in gerrit

Related Objects

Event Timeline

Krinkle created this task.Jan 20 2022, 4:38 PM
Restricted Application added a project: Platform Engineering. · View Herald TranscriptJan 20 2022, 4:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle renamed this task from Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1157)Function: MediaWiki\Deferred\LinksUpdate\LinksTable::doWritesQuery: DELETE FROM `page_props` WHERE (pp_page = 8898 AND pp_propname IN ('kartographer_frames','kartographer_links') ) to Deadlock found in Function: MediaWiki\Deferred\LinksUpdate\LinksTable::doWritesQuery: DELETE FROM `page_props`.Jan 20 2022, 4:38 PM
Krinkle renamed this task from Deadlock found in Function: MediaWiki\Deferred\LinksUpdate\LinksTable::doWritesQuery: DELETE FROM `page_props` to Deadlock found in Function: LinksTable::doWritesQuery: DELETE FROM `page_props`.
Krinkle removed a project: Platform Engineering.
Krinkle moved this task from Inbox, needs triage to Doing: Prio Interrupt on the Performance-Team board.
tstarling added a project: Maps (Kartographer).Jan 28 2022, 2:02 AM

Searching logstash from Jan 19 to now, I noticed that 668 out of 674 errors come from deleting the properties kartographer_frames and/or kartographer_links.

tstarling added a comment.Jan 28 2022, 3:07 AM

I was able to reproduce this locally. I imported https://en.wikivoyage.org/wiki/Aegina and its templates, then I edited one of the templates it used, then in the job queue debug log:

2022-01-28T14:01:07.836207+11:00 3464598 [DBQuery] MediaWiki\Deferred\LinksUpdate\LinksTable::doWrites [0.001s] localhost: DELETE FROM `page_props` WHERE (pp_page = 983 AND pp_propname IN ('kartographer_frames','kartographer_links') )

Probably something about the weird data flow in Module:Map confuses refreshLinks.

tstarling added a comment.Jan 28 2022, 3:19 AM

OK, sorry, it's not a Kartographer issue, it is indeed from the LinksUpdate refactor. New property values are compared to existing values by strict comparison, which doesn't work when an integer is given. The old code used array_diff_assoc() which implicitly converted to string before comparing.

gerritbot added a comment.Jan 28 2022, 4:07 AM

Change 757793 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] Fix page_props DELETE/INSERT on null edit

https://gerrit.wikimedia.org/r/757793

gerritbot added a project: Patch-For-Review.Jan 28 2022, 4:07 AM
tstarling removed a project: Maps (Kartographer).Jan 28 2022, 4:07 AM
tstarling claimed this task.Jan 31 2022, 12:29 AM
gerritbot added a comment.Jan 31 2022, 2:54 PM

Change 757793 merged by jenkins-bot:

[mediawiki/core@master] Fix page_props DELETE/INSERT on null edit

https://gerrit.wikimedia.org/r/757793

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.20; 2022-01-31).Jan 31 2022, 3:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 31 2022, 3:10 PM
Michael subscribed.Feb 1 2022, 5:15 PM
Umherirrender closed this task as Resolved.Feb 1 2022, 7:27 PM
• Nikerabbit added a project: affects-translatewiki.net.Feb 2 2022, 1:35 PM
• Nikerabbit subscribed.

We saw this during update.php call during deployment. I'll assume after we deploy the new version, it will have the fix and this issue will not appear again for us.

Krinkle moved this task from Untriaged to Jan 2022 on the Wikimedia-production-error board.Feb 4 2022, 12:44 AM
Krinkle mentioned this in T314184: Document the 2022-01-14 LinksUpdate incident.Jul 29 2022, 9:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits