Merge triggered warning about unequal numbers of ids to be restored and records to be inserted into master
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error message

PHP Warning: [data-update-failed]: A data update callback triggered an exception (Fail-safe exception. Number of ids to be restored is not equal to the number of records that are about to be inserted into master. This should never happen, except for an edge-case that was not detected during development or due to a race-condition that is not covered by this implementation.) [Called from Wikibase\Repo\Content\DataUpdateAdapter::doUpdate in /srv/mediawiki/php-1.34.0-wmf.23/extensions/Wikibase/repo/includes/Content/DataUpdateAdapter.php at line 65]

Impact

19 events since 2019-09-18T20:10:36
This seems to have started with 1.34.0-wmf.23 see https://logstash.wikimedia.org/app/kibana#/dashboard/AWra4yyim2VjIW0682f0?_g=h@e5f2e6c&_a=h@1fb6f9a

Notes

first event: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2019.09.18/mediawiki?id=AW1F_4RTZKA7RpirTZqk&_g=h@44136fa

Details

Request ID: XYTW9wpAICIAAJRvn8AAAACA
Request URL: https://www.wikidata.org/wiki/Special:MergeItems

Stack Trace

#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.34.0-wmf.23/includes/debug/MWDebug.php(333): trigger_error(string, integer)
#2 /srv/mediawiki/php-1.34.0-wmf.23/includes/debug/MWDebug.php(188): MWDebug::sendMessage(string, array, string, integer)
#3 /srv/mediawiki/php-1.34.0-wmf.23/includes/GlobalFunctions.php(1079): MWDebug::warning(string, integer, integer, string)
#4 /srv/mediawiki/php-1.34.0-wmf.23/extensions/Wikibase/lib/includes/Reporting/LogWarningExceptionHandler.php(28): wfLogWarning(string, integer)
#5 /srv/mediawiki/php-1.34.0-wmf.23/extensions/Wikibase/repo/includes/Content/DataUpdateAdapter.php(65): Wikibase\Lib\Reporting\LogWarningExceptionHandler->handleException(Exception, string, string)
#6 /srv/mediawiki/php-1.34.0-wmf.23/includes/deferred/DeferredUpdates.php(383): Wikibase\Repo\Content\DataUpdateAdapter->doUpdate()
#7 /srv/mediawiki/php-1.34.0-wmf.23/includes/Storage/DerivedPageDataUpdater.php(1630): DeferredUpdates::attemptUpdate(Wikibase\Repo\Content\DataUpdateAdapter, Wikimedia\Rdbms\LBFactoryMulti)
#8 /srv/mediawiki/php-1.34.0-wmf.23/includes/Storage/DerivedPageDataUpdater.php(1446): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(array)
#9 /srv/mediawiki/php-1.34.0-wmf.23/includes/deferred/MWCallableUpdate.php(38): MediaWiki\Storage\DerivedPageDataUpdater->MediaWiki\Storage\{closure}()
#10 /srv/mediawiki/php-1.34.0-wmf.23/includes/deferred/DeferredUpdates.php(383): MWCallableUpdate->doUpdate()
#11 /srv/mediawiki/php-1.34.0-wmf.23/includes/deferred/DeferredUpdates.php(281): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#12 /srv/mediawiki/php-1.34.0-wmf.23/includes/deferred/DeferredUpdates.php(226): DeferredUpdates::run(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#13 /srv/mediawiki/php-1.34.0-wmf.23/includes/deferred/DeferredUpdates.php(149): DeferredUpdates::handleUpdateQueue(array, string, integer)
#14 /srv/mediawiki/php-1.34.0-wmf.23/includes/MediaWiki.php(934): DeferredUpdates::doUpdates(string)
#15 /srv/mediawiki/php-1.34.0-wmf.23/includes/MediaWiki.php(758): MediaWiki->restInPeace(string, boolean)
#16 /srv/mediawiki/php-1.34.0-wmf.23/includes/MediaWiki.php(781): MediaWiki->{closure}()
#17 /srv/mediawiki/php-1.34.0-wmf.23/includes/MediaWiki.php(556): MediaWiki->doPostOutputShutdown(string)
#18 /srv/mediawiki/php-1.34.0-wmf.23/index.php(44): MediaWiki->run()
#19 /srv/mediawiki/w/index.php(3): require(string)
#20 {main}

Subject	Repo	Branch	Lines +/-
Failing test for T233414	mediawiki/extensions/Wikibase	master	+7 -3
Simply return ids to insert instead of failing	mediawiki/extensions/Wikibase	master	+7 -15
Temporary add metrics of the need to reinsert in the new term store	mediawiki/extensions/Wikibase	master	+6 -0
Temporary add metrics of the need to reinsert in the new term store	mediawiki/extensions/Wikibase	wmf/1.35.0-wmf.11	+6 -0

Customize query in gerrit

Related Objects

Mentioned In: T208425: [EPIC] Kill the wb_terms table
T151789: Wikibase\\TermSqlIndex::deleteTerms: Deadlock found when trying to get lock; try restarting transaction
T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; try restarting transaction

Event Timeline

Michael created this task.Sep 20 2019, 2:49 PM

Restricted Application added a project: Wikidata. · View Herald TranscriptSep 20 2019, 2:49 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Michael updated the task description. (Show Details)Sep 20 2019, 2:52 PM

• alaa_wmde added a project: User-Alaa.Sep 24 2019, 2:54 PM

• alaa_wmde moved this task from Backlog to Todo: 10 minutes thing on the User-Alaa board.

Krinkle moved this task from Untriaged to Sep2019/1.34.wmf.21+ on the Wikimedia-production-error board.Sep 24 2019, 7:39 PM

Seeing quite a bit of this in production logs at present:

brennen@mwlog1001:/srv/mw-log$ grep -c 'A data update callback triggered an exception' error.log
1018

Looks to still be happening

addshore@mwlog1001:/srv/mw-log$ grep -c 'A data update callback triggered an exception' error.log
304
addshore@mwlog1001:/srv/mw-log$ grep -c 'DataUpdateAdapter::doUpdate' error.log
304

Addshore triaged this task as Medium priority.Nov 19 2019, 12:56 PM

Addshore moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.

WMDE-leszek moved this task from Ready to estimate to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.Dec 3 2019, 1:14 PM

WMDE-leszek edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.

Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Dec 3 2019, 1:15 PM

Still happening: 240

Ladsgroup mentioned this in T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; try restarting transaction.Dec 10 2019, 11:15 AM

Ladsgroup merged a task: T237723: PHP Warning: [data-update-failed]: Fail-safe exception. Number of ids to be restored is not equal to the number of records that are about to be inserted into master.….) [Called from Wikibase/repo/includes/Content/DataUpdateAdapter.php].Dec 10 2019, 2:50 PM

Ladsgroup added subscribers: WMDE-leszek, • alaa_wmde.

I licked this.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptDec 11 2019, 11:50 PM

Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.Dec 12 2019, 12:15 AM

It's caused by this patch and that $recordsToInsert doesn't necessary have all of the $idsToRestore cases because some might be already in the table. Fixing it is not easy because $recordsToInsert is an associative array of terms properly mapped and $idsToRestore is just list of term ids. I leave this for anyone else to pick up for now.

Ladsgroup mentioned this in T151789: Wikibase\\TermSqlIndex::deleteTerms: Deadlock found when trying to get lock; try restarting transaction.Dec 17 2019, 9:54 PM

Jdforrester-WMF subscribed.Dec 18 2019, 3:31 PM

okay since no one touched it until I came back from vacation (I took vacation to run away from this :D) it seems I should do it.

Macro challenge-accepted:

Ladsgroup mentioned this in T208425: [EPIC] Kill the wb_terms table.Jan 6 2020, 7:54 AM

Change 562344 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Failing test for T233414

https://gerrit.wikimedia.org/r/562344

gerritbot added a project: Patch-For-Review.Jan 6 2020, 7:51 PM

Change 562475 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Temporary add metrics of the need to reinsert in the new term store

https://gerrit.wikimedia.org/r/562475

Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 7 2020, 9:26 AM

Change 562477 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.11] Temporary add metrics of the need to reinsert in the new term store

https://gerrit.wikimedia.org/r/562477

Ladsgroup moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 7 2020, 10:16 AM

Change 562475 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Temporary add metrics of the need to reinsert in the new term store

https://gerrit.wikimedia.org/r/562475

Change 562477 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.11] Temporary add metrics of the need to reinsert in the new term store

https://gerrit.wikimedia.org/r/562477

The graph: https://grafana.wikimedia.org/d/000000378/ladsgroup-test?orgId=1&fullscreen&panelId=2

Nothing so far.

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.14; 2020-01-07).Jan 7 2020, 11:00 AM

Change 562565 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Simply return ids to insert instead of failing

https://gerrit.wikimedia.org/r/562565

Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 7 2020, 6:02 PM

Change 562565 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Simply return ids to insert instead of failing

https://gerrit.wikimedia.org/r/562565

Ladsgroup moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 13 2020, 12:54 PM

ReleaseTaggerBot edited projects, added MW-1.35-notes (1.35.0-wmf.15; 2020-01-14); removed MW-1.35-notes (1.35.0-wmf.14; 2020-01-07).Jan 13 2020, 1:00 PM

Addshore closed this task as Resolved.Jan 15 2020, 5:07 PM

Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.Jan 15 2020, 5:15 PM

Change 562344 abandoned by Ladsgroup:
Failing test for T233414

Reason:
not needed anymore. Fixed.

https://gerrit.wikimedia.org/r/562344

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2020, 10:10 AM