Fail-safe exception from ReplicaMasterAwareRecordIdsAcquirer via Wikibase DataUpdateAdapter::doUpdate
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

MediaWiki version: 1.35.0-wmf.23

message

PHP Warning: [data-update-failed]: A data update callback triggered an exception (Fail-safe exception. Avoiding infinite loop due to possibily undetectable existing records in master.
 It may be due to encoding incompatibility between database values and values passed in $neededRecords parameter.) [Called from Wikibase\Repo\Content\DataUpdateAdapter::doUpdate in /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/repo/includes/Content/DataUpdateAdapter.php at line 62]

Impact

Notes

Details

Request ID: XmqKEQpAMNoAAhz8BDAAAAAW
Request URL: https://www.wikidata.org/w/api.php

Stack Trace

exception.trace

#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.35.0-wmf.23/includes/debug/MWDebug.php(333): trigger_error(string, integer)
#2 /srv/mediawiki/php-1.35.0-wmf.23/includes/debug/MWDebug.php(188): MWDebug::sendMessage(string, array, string, integer)
#3 /srv/mediawiki/php-1.35.0-wmf.23/includes/GlobalFunctions.php(1068): MWDebug::warning(string, integer, integer, string)
#4 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/Reporting/LogWarningExceptionHandler.php(28): wfLogWarning(string, integer)
#5 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/repo/includes/Content/DataUpdateAdapter.php(62): Wikibase\Lib\Reporting\LogWarningExceptionHandler->handleException(Exception, string, string)
#6 /srv/mediawiki/php-1.35.0-wmf.23/includes/deferred/DeferredUpdates.php(417): Wikibase\Repo\Content\DataUpdateAdapter->doUpdate()
#7 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/DerivedPageDataUpdater.php(1648): DeferredUpdates::attemptUpdate(Wikibase\Repo\Content\DataUpdateAdapter, Wikimedia\Rdbms\LBFactoryMulti)
#8 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/DerivedPageDataUpdater.php(1458): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(array)
#9 /srv/mediawiki/php-1.35.0-wmf.23/includes/deferred/MWCallableUpdate.php(38): MediaWiki\Storage\DerivedPageDataUpdater->MediaWiki\Storage\{closure}()
#10 /srv/mediawiki/php-1.35.0-wmf.23/includes/deferred/DeferredUpdates.php(417): MWCallableUpdate->doUpdate()
#11 /srv/mediawiki/php-1.35.0-wmf.23/includes/deferred/DeferredUpdates.php(296): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#12 /srv/mediawiki/php-1.35.0-wmf.23/includes/deferred/DeferredUpdates.php(233): DeferredUpdates::run(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#13 /srv/mediawiki/php-1.35.0-wmf.23/includes/deferred/DeferredUpdates.php(150): DeferredUpdates::handleUpdateQueue(array, string, integer)
#14 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(1058): DeferredUpdates::doUpdates(string)
#15 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(838): MediaWiki->restInPeace()
#16 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(847): MediaWiki->{closure}()
#17 /srv/mediawiki/php-1.35.0-wmf.23/api.php(119): MediaWiki->doPostOutputShutdown()
#18 /srv/mediawiki/w/api.php(3): require(string)
#19 {main}

Subject	Repo	Branch	Lines +/-
wbterms: When reaching an edge case, try reading master with CONN_TRX_AUTOCOMMIT	mediawiki/extensions/Wikibase	master	+30 -22
Do not lock rows when there's no term returned	mediawiki/extensions/Wikibase	wmf/1.35.0-wmf.24	+12 -2
Do not lock rows when there's no term returned	mediawiki/extensions/Wikibase	wmf/1.35.0-wmf.23	+12 -2
Do not lock rows when there's no term returned	mediawiki/extensions/Wikibase	master	+12 -2
Store item terms at late as possible to avoid deadlocks	mediawiki/extensions/Wikibase	master	+8 -5
Store item terms at late as possible to avoid deadlocks	mediawiki/extensions/Wikibase	wmf/1.35.0-wmf.23	+8 -5

Customize query in gerrit

Related Objects

Mentioned In: T252061: [data-update-failed]: Avoiding infinite loop due to possibily undetectable existing records (Wikibase DataUpdateAdapter)
T246898: Wikibase\Repo\Content\DataUpdateAdapter::doUpdate: Commit failed on server(s) 10.64.48.172: Cannot execute query from Wikibase\Repo\Content\DataUpdateAdapter::doUpdate while transaction status is ERROR
Mentioned Here: T246898: Wikibase\Repo\Content\DataUpdateAdapter::doUpdate: Commit failed on server(s) 10.64.48.172: Cannot execute query from Wikibase\Repo\Content\DataUpdateAdapter::doUpdate while transaction status is ERROR

Event Timeline

• nnikkhoui created this task.Mar 12 2020, 7:15 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 12 2020, 7:15 PM

• nnikkhoui added a parent task: T233871: 1.35.0-wmf.23 deployment blockers.Mar 12 2020, 7:16 PM

Train blocker -> UBN.

Restricted Application added a project: Wikidata. · View Herald TranscriptMar 12 2020, 7:19 PM

Restricted Application added a subscriber: Liuxinyu970226. · View Herald Transcript

Agusbou2015 subscribed.Mar 12 2020, 7:21 PM

Currently under discussion in #wikimedia-operations, seems likely a fix is inbound.

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/579378

On IRC it's suggested that this is a dupe of T246898, made worse by the config change.

This appears to have been mitigated by the above patch. Unclear whether this task should remain open, but removing as train blocker.

brennen removed a parent task: T233871: 1.35.0-wmf.23 deployment blockers.Mar 12 2020, 8:19 PM

Lowering the priority given this is not blocking a train any more.

Change 580302 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Store item terms at late as possible to avoid deadlocks

https://gerrit.wikimedia.org/r/580302

gerritbot added a project: Patch-For-Review.Mar 17 2020, 11:18 AM

Ladsgroup claimed this task.Mar 17 2020, 1:12 PM

Ladsgroup edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptMar 17 2020, 1:12 PM

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 17 2020, 1:13 PM

Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Mar 17 2020, 1:15 PM

Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.

Change 580328 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.23] Store item terms at late as possible to avoid deadlocks

https://gerrit.wikimedia.org/r/580328

Change 580302 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Store item terms at late as possible to avoid deadlocks

https://gerrit.wikimedia.org/r/580302

Change 580328 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.23] Store item terms at late as possible to avoid deadlocks

https://gerrit.wikimedia.org/r/580328

Mentioned in SAL (#wikimedia-operations) [2020-03-17T14:14:34Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseItemTermStoreWriter.php: [[gerrit:580328|Store item terms at late as possible to avoid deadlocks (T247553 T246898)]] (duration: 01m 07s)

Stashbot mentioned this in T246898: Wikibase\Repo\Content\DataUpdateAdapter::doUpdate: Commit failed on server(s) 10.64.48.172: Cannot execute query from Wikibase\Repo\Content\DataUpdateAdapter::doUpdate while transaction status is ERROR.Mar 17 2020, 2:14 PM

Change 580342 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Do not lock rows when there's no term returned

https://gerrit.wikimedia.org/r/580342

Change 580352 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.23] Do not lock rows when there's no term returned

https://gerrit.wikimedia.org/r/580352

Change 580342 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Do not lock rows when there's no term returned

https://gerrit.wikimedia.org/r/580342

Change 580390 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.24] Do not lock rows when there's no term returned

https://gerrit.wikimedia.org/r/580390

Change 580352 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.23] Do not lock rows when there's no term returned

https://gerrit.wikimedia.org/r/580352

Mentioned in SAL (#wikimedia-operations) [2020-03-17T17:46:05Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseItemTermStoreWriter.php: [[gerrit:580352|Do not lock rows when there's no term returned (T247553 T246898)]] (duration: 01m 07s)

Ladsgroup moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 17 2020, 6:03 PM

Change 580390 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.24] Do not lock rows when there's no term returned

https://gerrit.wikimedia.org/r/580390

Mentioned in SAL (#wikimedia-operations) [2020-03-17T18:53:11Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.24/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseItemTermStoreWriter.php: [[gerrit:580390|Do not lock rows when there's no term returned (T247553 T246898)]], To catch the train (duration: 01m 08s)

Maintenance_bot removed a project: Patch-For-Review.Mar 17 2020, 7:10 PM

Krinkle renamed this task from PHP Warning: [data-update-failed]: A data update callback triggered an exception (Fail-safe exception. Avoiding infinite loop due to possibily undetectable existing records in master. It may be due to encoding incompatibility between database values and values passed in $neededRecords parameter.) [Called from Wikibase\Repo\Content\DataUpdateAdapter::doUpdate in /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/repo/includes/Content/DataUpdateAdapter.php at line 62] to Fail-safe exception from ReplicaMasterAwareRecordIdsAcquirer via Wikibase DataUpdateAdapter::doUpdate.Mar 18 2020, 3:45 PM

Krinkle moved this task from Untriaged to Mar 2020 on the Wikimedia-production-error board.

This is actually two different errors. I make another one for the new one.

Ladsgroup moved this task from Test (Verification) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 18 2020, 3:56 PM

Addshore subscribed.Mar 19 2020, 11:03 AM

I think I found out what's happening here. I tracked down one case of it happening which is this edit: https://www.wikidata.org/w/index.php?diff=1138568411 causing this: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-mediawiki-2020.03.19/mediawiki/?id=AXDzmOx6KWrIH1QRGQJO

Why? This edit happening almost at the same time: https://www.wikidata.org/w/index.php?diff=1138568386

Both edits introduce a new text ("Biblioteca Pública Municipal no município de Muquém de São Francisco") for the term store at the same time and the code tries to read it, it can't (because it doesn't exist yet) then it tries to insert it, it can't (because the other thread already put it there) then it tries to read it again (because it couldn't insert, right?) but it can't find it which seems like a very logical thing to do to throw a logic error (narrator: it's not).

The reason that the second read can't read the ghost row is MySQL's default locking level: "REPEATABLE_READ". I guess that's explanatory enough. when you read from master from one transaction, MySQL gives you the old version due to MVCC but it doesn't let you insert it either.

I don't know what would be the right solution right now but I will find it.

Change 582159 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] wbterms: When reaching an edge case, try reading master with CONN_TRX_AUTOCOMMIT

https://gerrit.wikimedia.org/r/582159

gerritbot added a project: Patch-For-Review.Mar 20 2020, 7:47 PM

Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 20 2020, 7:49 PM

hoo moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 23 2020, 12:23 PM

Change 582159 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] wbterms: When reaching an edge case, try reading master with CONN_TRX_AUTOCOMMIT

https://gerrit.wikimedia.org/r/582159

Maintenance_bot removed a project: Patch-For-Review.Mar 23 2020, 2:10 PM

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.25; 2020-03-24).Mar 23 2020, 9:00 PM

Addshore closed this task as Resolved.May 4 2020, 1:04 PM

Addshore moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.May 4 2020, 1:15 PM

brennen mentioned this in T252061: [data-update-failed]: Avoiding infinite loop due to possibily undetectable existing records (Wikibase DataUpdateAdapter).May 6 2020, 7:34 PM