Maniphest T204065

WikibaseLexeme CI broken (database errors)
Closed, ResolvedPublic
Actions

Description

For the past few hours, CI on Wikibase-related repositories has been mostly broken (both test builds and gate-and-submit). Some example changes:

Some example consoles:

They all seem to have similar error outputs: first some NameTableAccessExceptions…

Expected unused ID from database insert for 'Scribunto' into 'content_models', but ID 2 is already associated with the name 'wikibase-item'! This may indicate database corruption!

… and then some query errors:

Wikimedia\Rdbms\DBTransactionStateError: Cannot execute query from MessageCache::loadFromDB(en)-big while transaction status is ERROR.
…
Caused by
Wikimedia\Rdbms\DBUnexpectedError: Uncancelable atomic section canceled (got MediaWiki\Storage\RevisionStore::insertRevisionOn).
…
Warning: Destructor threw an object exception: exception 'Wikimedia\Rdbms\DBUnexpectedError' with message 'Uncancelable atomic section canceled (got MediaWiki\Storage\RevisionStore::insertRevisionOn).' in /workspace/src/includes/libs/rdbms/database/Database.php:3735

Details

Subject	Repo	Branch	Lines +/-
DNM: check that Lexeme CI passes in MCR read-new mode.	mediawiki/extensions/WikibaseLexeme	master	+3 -0
Add WikibaseLexemeExtensionRegistrationTest to group Database	mediawiki/extensions/WikibaseLexeme	master	+2 -0
Reset services in ParserTestTopLevelSuite.	mediawiki/core	master	+112 -54
Revert MCR migration stage to write-both/read-old	mediawiki/core	master	+1 -1
Only setup DB in tests that say they need the DB	mediawiki/core	master	+1 -1
[DNM] empty change to test CI	mediawiki/extensions/WikibaseQualityConstraints	master	+2 -1

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		daniel	T204065 WikibaseLexeme CI broken (database errors)
Resolved		Addshore	T204153 Add more Wikibase extensions to gatedextensions
Open		None	T204252 Have dependencies of gated extensions in the gate
Resolved	PRODUCTION ERROR	Reedy	T198653 Add AntiSpoof to shared extension gate
Duplicate		None	T200976 Wikibase CI: wmf-quibble-vendor-mysql-hhvm-docker job should include Scribunto
Resolved		Jdforrester-WMF	T125050 [Task] Add Scribunto to extension-gate in CI
Resolved		Jdforrester-WMF	T126670 Have CI set `$wgScribuntoDefaultEngine = 'luasandbox` to speed up parser tests
Resolved		None	T128191 Wikibase Scribunto tests are failing, with the tests depending on configuration
Resolved		hoo	T128245 Use $this->getEngine() instead of Scribunto::newDefaultEngine()
Resolved		• chasemp	T162166 For contintcloud either add RAM or Swap to the instances
Resolved		Jdforrester-WMF	T225068 Add a PHPUnit group to skip test on gated CI runs
Resolved		hashar	T202030 CI: Minerva PHPUnit tests should be included in shared extension gate job
Declined		None	T321864 Add CentralAuth to the gated extensions list

Event Timeline

Lucas_Werkmeister_WMDE created this task.Sep 11 2018, 3:36 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 11 2018, 3:36 PM

Addshore added projects: Multi-Content-Revisions, User-Addshore.Sep 11 2018, 3:38 PM

Addshore added subscribers: Addshore, daniel.

Lucas_Werkmeister_WMDE updated the task description. (Show Details)Sep 11 2018, 3:38 PM

Looking at core merges recently it must be one of these 2 patches:

Introduce RevisionRecord::isReadForInsertion https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/459620/
[MCR] Set MCR migration stage to write-both/read-new. … https://gerrit.wikimedia.org/r/#/c/443831/

Perhaps core should be running against WikibaseLexeme? at least on gate? I thought it should have been...

Addshore triaged this task as High priority.Sep 11 2018, 3:41 PM

Addshore raised the priority of this task from High to Unbreak Now!.Sep 11 2018, 3:46 PM

Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptSep 11 2018, 3:46 PM

Addshore added subscribers: hashar, • Pablo-WMDE, WMDE-leszek.Sep 11 2018, 3:48 PM

The NameTableAccessException feels like it could be happening because the content_models table is purged between two tests, but the NameTableStore’s cache of it isn’t purged. I had a similar problem in WikibaseQualityConstraints (fixed in I8cdebc0eef).

In T204065#4574491, @Lucas_Werkmeister_WMDE wrote:

The NameTableAccessException feels like it could be happening because the content_models table is purged between two tests, but the NameTableStore’s cache of it isn’t purged. I had a similar problem in WikibaseQualityConstraints (fixed in I8cdebc0eef).

Yup, @daniel confirmed that is likely what is happening.

So @Addshore seems to be on it.

Addshore assigned this task to daniel.Sep 11 2018, 4:07 PM

Change 459812 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/core@master] [DNM] Set MCR migration stage to write-both/read-old

https://gerrit.wikimedia.org/r/459812

Change 459813 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/WikibaseQualityConstraints@master] [DNM] empty change to test CI

https://gerrit.wikimedia.org/r/459813

Change 459813 abandoned by Lucas Werkmeister (WMDE):
[DNM] empty change to test CI

Reason:
CI passed.

https://gerrit.wikimedia.org/r/459813

Investigation is still going on, but it looks like reverting the MCR migration stage would be a simple way to temporarily unbreak this, if desired. See I5426b5efd0 (but change the commit message before merging that, of course ☺)

Change 459820 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] Reset the NameTableStoreFactory service in ParserTestRunner

https://gerrit.wikimedia.org/r/459820

Change 459821 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/WikibaseLexeme@master] Add WikibaseLexemeExtensionRegistrationTest to group Database

https://gerrit.wikimedia.org/r/459821

Change 459823 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/core@master] Only setup DB in tests that say they need the DB

https://gerrit.wikimedia.org/r/459823

Addshore mentioned this in rEWLE8d37e3f3e23b: Add WikibaseLexemeExtensionRegistrationTest to group Database.Sep 11 2018, 5:09 PM

Change 459823 abandoned by Addshore:
Only setup DB in tests that say they need the DB

Reason:
abandoning as there is an identical path

https://gerrit.wikimedia.org/r/459823

daniel mentioned this in rEWLE84e61dfc925b: Add WikibaseLexemeExtensionRegistrationTest to group Database.Sep 11 2018, 5:20 PM

daniel mentioned this in T198561: Make "write both, read new" the default MCR migration stage for fresh MediaWiki installs / for CI.Sep 11 2018, 5:55 PM

Change 459812 merged by jenkins-bot:
[mediawiki/core@master] Revert MCR migration stage to write-both/read-old

https://gerrit.wikimedia.org/r/459812

So after those reverts it's no longer broken, correct? Resetting priority based on that assumption. I'll leave you all to determine follow-ups/next steps.

In T204065#4575049, @greg wrote:

So after those reverts it's no longer broken, correct? Resetting priority based on that assumption. I'll leave you all to determine follow-ups/next steps.

The "fix" has derailed MCR/SDC work. Not sure it's not still UBN…

Sorry, I couldn't tell from this task what the issue is; is it CI? If so in what way? Is it the change/how things are being done? If so how? :) Just trying to make informed decisions :)

Switching MCR in master to read-new (T198561) broke a bunch of the Wikibase-related extensions' unit tests (and thus they couldn't merge anything, which is sad). Now we're back to read-old in master which has unbroken Wikibase and means the MCR team will need to fix Wikibase and restore before they can proceed.

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)).Sep 11 2018, 7:00 PM

JeroenDeDauw mentioned this in rEWLE4d94cd779c3c: Add WikibaseLexemeExtensionRegistrationTest to group Database.Sep 12 2018, 7:19 AM

So, the CI is now fixed.
As a follow up it could be a good idea to add WikibaseLexeme to the list of gated extensions for core @greg @hashar ?

Change 459991 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] Add more Wikibase extensions to gatedextensions

https://gerrit.wikimedia.org/r/459991

Change 460017 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/WikibaseLexeme@master] DNM: check that Lexeme CI passes in MCR read-new mode.

https://gerrit.wikimedia.org/r/460017

daniel mentioned this in rEWLE59f128a6eccd: DNM: check that Lexeme CI passes in MCR read-new mode..Sep 12 2018, 1:36 PM

daniel mentioned this in rEWLE3f7ae684abaa: DNM: check that Lexeme CI passes in MCR read-new mode..Sep 12 2018, 1:40 PM

daniel mentioned this in rEWLE3b3828ff9a7b: DNM: check that Lexeme CI passes in MCR read-new mode..

Jdforrester-WMF renamed this task from Wikibase CI broken (database errors) to WikibaseLexeme CI broken (database errors).Sep 12 2018, 3:24 PM

Change 459820 merged by jenkins-bot:
[mediawiki/core@master] Reset services in ParserTestTopLevelSuite.

https://gerrit.wikimedia.org/r/459820

Change 459821 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] Add WikibaseLexemeExtensionRegistrationTest to group Database

https://gerrit.wikimedia.org/r/459821

daniel closed this task as Resolved.Sep 12 2018, 6:59 PM

In T204065#4576329, @Addshore wrote:

So, the CI is now fixed.
As a follow up it could be a good idea to add WikibaseLexeme to the list of gated extensions for core @greg @hashar ?

Filled as T204153

hashar mentioned this in T204252: Have dependencies of gated extensions in the gate.Sep 13 2018, 3:19 PM

Change 460017 abandoned by Jforrester:
DNM: check that Lexeme CI passes in MCR read-new mode.

Reason:
Testing over, this has landed.

https://gerrit.wikimedia.org/r/460017

Liuxinyu970226 unsubscribed.Sep 14 2018, 1:12 PM

CCicalese_WMF moved this task from Inbox to Done on the Multi-Content-Revisions board.Sep 18 2018, 10:09 PM

Addshore moved this task from Unsorted 💣 to Closing ✔️ on the User-Addshore board.Sep 20 2018, 1:10 PM

Addshore closed subtask T204153: Add more Wikibase extensions to gatedextensions as Resolved.Jan 22 2021, 9:26 PM

Restricted Application added a project: [DEPRECATED] wdwb-tech. · View Herald TranscriptJan 22 2021, 9:26 PM