Page MenuHomePhabricator

Deadlock in SiteLinkTable::insertLinks on Item merges
Open, LowPublicPRODUCTION ERROR

Description

Warning: [data-update-failed]: A data update callback triggered an exception (A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? Query: INSERT IGNORE INTO wb_items_per_site (ips_item_id,ips_site_id,ips_site_page) VALUES ('346509','ukwiki','Нитка (значення)') Function: Wikibase\Lib\Store\SiteLinkTable::insertLinks Error: 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.16.144) ) [Called from Wikibase\Updates\DataUpdateAdapter::doUpdate in /srv/mediawiki/php-1.28.0-wmf.22/extensions/Wikidata/extensions/Wikibase/repo/includes/updates/DataUpdateAdapter.php at line 73] in /srv/mediawiki/php-1.28.0-wmf.22/includes/debug/MWDebug.php on line 311

See also: T151789

Event Timeline

hoo triaged this task as Low priority.Oct 12 2016, 10:32 PM

Doesn't seem to be very common:

$ zgrep -c 'Deadlock.*SiteLinkTable::insertLinks' archive/DBQuery.log-2016*
archive/DBQuery.log-20160923.gz:1
archive/DBQuery.log-20160924.gz:1
archive/DBQuery.log-20160925.gz:0
archive/DBQuery.log-20160926.gz:1
archive/DBQuery.log-20160927.gz:2
archive/DBQuery.log-20160928.gz:0
archive/DBQuery.log-20160929.gz:0
archive/DBQuery.log-20160930.gz:0
archive/DBQuery.log-20161001.gz:1
archive/DBQuery.log-20161002.gz:0
archive/DBQuery.log-20161003.gz:0
archive/DBQuery.log-20161004.gz:0
archive/DBQuery.log-20161005.gz:0
archive/DBQuery.log-20161006.gz:0
archive/DBQuery.log-20161007.gz:0
archive/DBQuery.log-20161008.gz:1
archive/DBQuery.log-20161009.gz:0
archive/DBQuery.log-20161010.gz:1
archive/DBQuery.log-20161011.gz:1
archive/DBQuery.log-20161012:1
hoo renamed this task from [BUG] wiki base fails to update DB to Deadlock in SiteLinkTable::insertLinks on Item merges.Oct 12 2016, 11:05 PM

I checked a few cases here and this seems to be solely a problem with Item merges. I guess this happens when we enqueue the post-Edit updates and the one for the to Item and the one of the from Item are run at the same time.

Sadly still (very occasionally) happening (10 times this year so far):

$ zgrep -c 'Deadlock.*SiteLinkTable::insertLinks' archive/DBQuery.log-2017*
archive/DBQuery.log-20170101.gz:0
archive/DBQuery.log-20170102.gz:0
archive/DBQuery.log-20170103.gz:1
archive/DBQuery.log-20170104.gz:3
archive/DBQuery.log-20170105.gz:2
archive/DBQuery.log-20170106.gz:0
archive/DBQuery.log-20170107.gz:0
archive/DBQuery.log-20170108.gz:2
archive/DBQuery.log-20170109.gz:1
archive/DBQuery.log-20170110:1
Krinkle updated the task description. (Show Details)
Krinkle changed Risk Rating from N/A to default.
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM

It might help if we make sure to insert and delete rows in a consistent order (ordered by site ID)… should be easy enough for insertLinks() and deleteLinks(), but deleteLinksOfItem() is going to be slightly trickier.

SiteLinkTable.php
	public function deleteLinksOfItem( ItemId $itemId ) {
		if ( $this->readonly ) {
			throw new MWException( 'Cannot write when in readonly mode' );
		}

		$dbw = $this->db->connections()->getWriteConnectionRef();

		$dbw->delete(
			$this->table,
			[ 'ips_item_id' => $itemId->getNumericId() ],
			__METHOD__
		);

		return true;
	}

We don’t have the list of links here, we’re just deleting everything for the item. I’d have to look up if IDatabase supports deleting according to a subquery (select the sitelinks with that item ID with an ORDER BY).

(Or we could just speculate that deleteLinksOfItem() deletes in index order, and if we sort the other two queries by site ID then it works out on its own?)

Actually, it looks like this deadlock is no longer happening at all:

lucaswerkmeister-wmde@mwlog1002:~$ zgrep -c 'Deadlock.*SiteLinkTable' /srv/mw-log/archive/DBQuery.log-2021*
/srv/mw-log/archive/DBQuery.log-20210622.gz:0
/srv/mw-log/archive/DBQuery.log-20210623.gz:0
/srv/mw-log/archive/DBQuery.log-20210624.gz:0
/srv/mw-log/archive/DBQuery.log-20210625.gz:0
/srv/mw-log/archive/DBQuery.log-20210626.gz:0
/srv/mw-log/archive/DBQuery.log-20210627.gz:0
/srv/mw-log/archive/DBQuery.log-20210628.gz:0
/srv/mw-log/archive/DBQuery.log-20210629.gz:0
/srv/mw-log/archive/DBQuery.log-20210630.gz:0
/srv/mw-log/archive/DBQuery.log-20210701.gz:0
/srv/mw-log/archive/DBQuery.log-20210702.gz:0
/srv/mw-log/archive/DBQuery.log-20210703.gz:0
/srv/mw-log/archive/DBQuery.log-20210704.gz:0
/srv/mw-log/archive/DBQuery.log-20210705.gz:0
/srv/mw-log/archive/DBQuery.log-20210706.gz:0
/srv/mw-log/archive/DBQuery.log-20210707.gz:0
/srv/mw-log/archive/DBQuery.log-20210708.gz:0
/srv/mw-log/archive/DBQuery.log-20210709.gz:0
/srv/mw-log/archive/DBQuery.log-20210710.gz:0
/srv/mw-log/archive/DBQuery.log-20210711.gz:0
/srv/mw-log/archive/DBQuery.log-20210712.gz:0
/srv/mw-log/archive/DBQuery.log-20210713.gz:0
/srv/mw-log/archive/DBQuery.log-20210714.gz:0
/srv/mw-log/archive/DBQuery.log-20210715.gz:0
/srv/mw-log/archive/DBQuery.log-20210716.gz:0
/srv/mw-log/archive/DBQuery.log-20210717.gz:0
/srv/mw-log/archive/DBQuery.log-20210718.gz:0
/srv/mw-log/archive/DBQuery.log-20210719.gz:0
/srv/mw-log/archive/DBQuery.log-20210720.gz:0
/srv/mw-log/archive/DBQuery.log-20210721.gz:0
/srv/mw-log/archive/DBQuery.log-20210722.gz:0
/srv/mw-log/archive/DBQuery.log-20210723.gz:0
/srv/mw-log/archive/DBQuery.log-20210724.gz:0
/srv/mw-log/archive/DBQuery.log-20210725.gz:0
/srv/mw-log/archive/DBQuery.log-20210726.gz:0
/srv/mw-log/archive/DBQuery.log-20210727.gz:0
/srv/mw-log/archive/DBQuery.log-20210728.gz:0
/srv/mw-log/archive/DBQuery.log-20210729.gz:0
/srv/mw-log/archive/DBQuery.log-20210730.gz:0
/srv/mw-log/archive/DBQuery.log-20210731.gz:0
/srv/mw-log/archive/DBQuery.log-20210801.gz:0
/srv/mw-log/archive/DBQuery.log-20210802.gz:0
/srv/mw-log/archive/DBQuery.log-20210803.gz:0
/srv/mw-log/archive/DBQuery.log-20210804.gz:0
/srv/mw-log/archive/DBQuery.log-20210805.gz:0
/srv/mw-log/archive/DBQuery.log-20210806.gz:0
/srv/mw-log/archive/DBQuery.log-20210807.gz:0
/srv/mw-log/archive/DBQuery.log-20210808.gz:0
/srv/mw-log/archive/DBQuery.log-20210809.gz:0
/srv/mw-log/archive/DBQuery.log-20210810.gz:0
/srv/mw-log/archive/DBQuery.log-20210811.gz:0
/srv/mw-log/archive/DBQuery.log-20210812.gz:0
/srv/mw-log/archive/DBQuery.log-20210813.gz:0
/srv/mw-log/archive/DBQuery.log-20210814.gz:0
/srv/mw-log/archive/DBQuery.log-20210815.gz:0
/srv/mw-log/archive/DBQuery.log-20210816.gz:0
/srv/mw-log/archive/DBQuery.log-20210817.gz:0
/srv/mw-log/archive/DBQuery.log-20210818.gz:0
/srv/mw-log/archive/DBQuery.log-20210819.gz:0
/srv/mw-log/archive/DBQuery.log-20210820.gz:0
/srv/mw-log/archive/DBQuery.log-20210821.gz:0
/srv/mw-log/archive/DBQuery.log-20210822.gz:0
/srv/mw-log/archive/DBQuery.log-20210823.gz:0
/srv/mw-log/archive/DBQuery.log-20210824.gz:0
/srv/mw-log/archive/DBQuery.log-20210825.gz:0
/srv/mw-log/archive/DBQuery.log-20210826.gz:0
/srv/mw-log/archive/DBQuery.log-20210827.gz:0
/srv/mw-log/archive/DBQuery.log-20210828.gz:0
/srv/mw-log/archive/DBQuery.log-20210829.gz:0
/srv/mw-log/archive/DBQuery.log-20210830.gz:0
/srv/mw-log/archive/DBQuery.log-20210831.gz:0
/srv/mw-log/archive/DBQuery.log-20210901.gz:0
/srv/mw-log/archive/DBQuery.log-20210902.gz:0
/srv/mw-log/archive/DBQuery.log-20210903.gz:0
/srv/mw-log/archive/DBQuery.log-20210904.gz:0
/srv/mw-log/archive/DBQuery.log-20210905.gz:0
/srv/mw-log/archive/DBQuery.log-20210906.gz:0
/srv/mw-log/archive/DBQuery.log-20210907.gz:0
/srv/mw-log/archive/DBQuery.log-20210908.gz:0
/srv/mw-log/archive/DBQuery.log-20210909.gz:0
/srv/mw-log/archive/DBQuery.log-20210910.gz:0
/srv/mw-log/archive/DBQuery.log-20210911.gz:0
/srv/mw-log/archive/DBQuery.log-20210912.gz:0
/srv/mw-log/archive/DBQuery.log-20210913.gz:0
/srv/mw-log/archive/DBQuery.log-20210914.gz:0
/srv/mw-log/archive/DBQuery.log-20210915.gz:0
/srv/mw-log/archive/DBQuery.log-20210916.gz:0
/srv/mw-log/archive/DBQuery.log-20210917.gz:0
/srv/mw-log/archive/DBQuery.log-20210918.gz:0
/srv/mw-log/archive/DBQuery.log-20210919.gz:0
/srv/mw-log/archive/DBQuery.log-20210920.gz:0
/srv/mw-log/archive/DBQuery.log-20210921:0

There are still some other deadlocks, including in Wikibase, but they’re in other classes (EntityUsageTable is a popular one).