Wikimedia\Rdbms\DBQueryError: Error 1049: Unknown database 'cognate_wiktionary'
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

mwversion: 1.43.0-wmf.5
reqId: c24b6625-e3dd-4faf-84f5-942bf45f9739
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1049: Unknown database 'cognate_wiktionary'
Function: Wikimedia\Rdbms\DatabaseMySQL::doSelectDomain
Query: USE `cognate_wiktionary`

exception.trace

from /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/Database.php(1203)
#0 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/Database.php(1187): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/Database.php(1161): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/DatabaseMySQL.php(204): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string)
#3 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/Database.php(1514): Wikimedia\Rdbms\DatabaseMySQL->doSelectDomain(Wikimedia\Rdbms\DatabaseDomain)
#4 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/loadbalancer/LoadBalancer.php(912): Wikimedia\Rdbms\Database->selectDomain(Wikimedia\Rdbms\DatabaseDomain)
#5 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/loadbalancer/LoadBalancer.php(801): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(integer, Wikimedia\Rdbms\DatabaseDomain, integer)
#6 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/loadbalancer/LoadBalancer.php(793): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#7 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/DBConnRef.php(110): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#8 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/DBConnRef.php(124): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#9 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/DBConnRef.php(358): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#10 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(730): Wikimedia\Rdbms\DBConnRef->select(array, array, array, string, array, array)
#11 /srv/mediawiki/php-1.43.0-wmf.5/extensions/Cognate/src/CognateStore.php(175): Wikimedia\Rdbms\SelectQueryBuilder->fetchResultSet()
#12 /srv/mediawiki/php-1.43.0-wmf.5/extensions/Cognate/src/CognateRepo.php(140): Cognate\CognateStore->selectLinkDetailsForPage(string, MediaWiki\Title\Title)
#13 /srv/mediawiki/php-1.43.0-wmf.5/extensions/Cognate/src/HookHandler/CognateParseHookHandler.php(79): Cognate\CognateRepo->getLinksForPage(string, MediaWiki\Title\Title)
#14 /srv/mediawiki/php-1.43.0-wmf.5/extensions/Cognate/src/CognateHooks.php(125): Cognate\HookHandler\CognateParseHookHandler->doContentAlterParserOutput(MediaWiki\Title\Title, MediaWiki\Parser\ParserOutput)
#15 /srv/mediawiki/php-1.43.0-wmf.5/includes/HookContainer/HookContainer.php(159): Cognate\CognateHooks->onContentAlterParserOutput(WikitextContent, MediaWiki\Title\Title, MediaWiki\Parser\ParserOutput)
#16 /srv/mediawiki/php-1.43.0-wmf.5/includes/HookContainer/HookRunner.php(1185): MediaWiki\HookContainer\HookContainer->run(string, array)
#17 /srv/mediawiki/php-1.43.0-wmf.5/includes/content/ContentHandler.php(1690): MediaWiki\HookContainer\HookRunner->onContentAlterParserOutput(WikitextContent, MediaWiki\Title\Title, MediaWiki\Parser\ParserOutput)
#18 /srv/mediawiki/php-1.43.0-wmf.5/includes/content/Renderer/ContentRenderer.php(67): ContentHandler->getParserOutput(WikitextContent, MediaWiki\Content\Renderer\ContentParseParams)
#19 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiParse.php(161): MediaWiki\Content\Renderer\ContentRenderer->getParserOutput(WikitextContent, MediaWiki\Title\Title, NULL, ParserOptions)
#20 /srv/mediawiki/php-1.43.0-wmf.5/includes/poolcounter/PoolCounterWorkViaCallback.php(73): ApiParse->{closure}()
#21 /srv/mediawiki/php-1.43.0-wmf.5/includes/poolcounter/PoolCounterWork.php(171): MediaWiki\PoolCounter\PoolCounterWorkViaCallback->doWork()
#22 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiParse.php(169): MediaWiki\PoolCounter\PoolCounterWork->execute()
#23 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiParse.php(446): ApiParse->getContentParserOutput(WikitextContent, MediaWiki\Title\Title, NULL, ParserOptions)
#24 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiMain.php(1952): ApiParse->execute()
#25 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiMain.php(928): ApiMain->executeAction()
#26 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiMain.php(899): ApiMain->executeActionWithErrorHandling()
#27 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiEntryPoint.php(158): ApiMain->execute()
#28 /srv/mediawiki/php-1.43.0-wmf.5/includes/MediaWikiEntryPoint.php(199): MediaWiki\Api\ApiEntryPoint->execute()
#29 /srv/mediawiki/php-1.43.0-wmf.5/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#30 /srv/mediawiki/w/api.php(3): require(string)
#31 {main}

Impact

Logspam, and user scripts querying the aawiktionary API don’t work.

Notes

aawiktionary is a closed wiki, I don’t know if that’s relevant.
As far as I understand, cognate_wiktionary is a database that’s shared across several wikis.

Details

Request URL: https://aa.wiktionary.org/w/api.php?action=parse&format=json&prop=langlinks&text=&title=test

Subject	Repo	Branch	Lines +/-
rdbms: Regression tests for T364827	mediawiki/core	master	+16 -0
rdbms: Fix picking the database from the LB domain	mediawiki/core	master	+4 -2
rdbms: Fix picking the database from the LB domain	mediawiki/core	wmf/1.43.0-wmf.5	+4 -2

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	hashar	T361399 1.43.0-wmf.5 deployment blockers
		Resolved	PRODUCTION ERROR	Ladsgroup	T364827 Wikimedia\Rdbms\DBQueryError: Error 1049: Unknown database 'cognate_wiktionary'

Event Timeline

Lucas_Werkmeister_WMDE created this task.May 14 2024, 9:37 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2024, 9:37 AM

Lucas_Werkmeister_WMDE updated the task description. (Show Details)May 14 2024, 9:38 AM

From that error, db1175 belongs to s3, which doesn't store cognate_wiktionary - that database lives in x1.

aawiktionary is a closed wiki, I don’t know if that’s relevant.

Doesn’t seem like it, I can also reproduce it on dewiktionary as long as the &title= doesn’t exist. (aawiktionary then just has the special property that virtually every title is guaranteed to not exist.) edit: wrong, see below

hashar mentioned this in T364828: Wikimedia\Rdbms\DBQueryError: Error 1049: Unknown database 'wikishared'Function: Wikimedia\Rdbms\DatabaseMySQL::doSelectDomainQuery: USE `wikishared`.May 14 2024, 9:42 AM

Possibly related to T348526: [COG] [TECH] Migrate Cognate to use a virtual database domain? But given T364828, it seems more likely that something’s generally broken about shared (x1) databases.

hashar triaged this task as Unbreak Now! priority.May 14 2024, 9:44 AM

In T364827#9793637, @Lucas_Werkmeister_WMDE wrote:

I can also reproduce it on dewiktionary

This sounds like it’s not a train issue, by the way, since dewiktionary (group1) is still on wmf.4… unless it’s somehow loading wmf.5 code anyway? (The timing does feel suspicious, I’ll admit that.) edit: wrong, see below

hashar added a parent task: T361399: 1.43.0-wmf.5 deployment blockers.May 14 2024, 9:46 AM

In T364827#9793685, @Lucas_Werkmeister_WMDE wrote:

In T364827#9793637, @Lucas_Werkmeister_WMDE wrote:

I can also reproduce it on dewiktionary

This sounds like it’s not a train issue, by the way, since dewiktionary (group1) is still on wmf.4… unless it’s somehow loading wmf.5 code anyway? (The timing does feel suspicious, I’ll admit that.)

Scratch that, I somehow failed to actually change the URL to dewiktionary. Cannot reproduce the issue on dewiktionary after all, so it probably is the train.

I think I know what's going on. Give me a bit.

This is very likely caused by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1024973/3/includes/libs/rdbms/lbfactory/LBFactory.php a refactor we did on virtual domains, but I can't see how. I'm not spotting any obvious mistakes (I dig deeper). If you do, please let me know.

All of these errors are clearly because virtual domains is now broken. cc @Tgr I debug and see what I can find.

Ah found it. Sneaky bug. I patch it now.

Ladsgroup moved this task from Triage to In progress on the DBA board.May 14 2024, 10:12 AM

Change #1031410 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Fix picking the database from the LB domain

https://gerrit.wikimedia.org/r/1031410

gerritbot added a project: Patch-For-Review.May 14 2024, 10:17 AM

@Ladsgroup can you consider rolling back the code from MediaWiki and repurpose it for next week with test coverage added? I'd like to avoid monkey patching code :-)

That code is used in other places now, making the roll back more complicated. The fix is straightforward.

Maintenance_bot added a project: Wikidata.May 14 2024, 10:30 AM

Restricted Application added a project: wmde-wikidata-tech. · View Herald TranscriptMay 14 2024, 10:30 AM

Change #1031414 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Regression tests for T364827

https://gerrit.wikimedia.org/r/1031414

Change #1031177 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.43.0-wmf.5] rdbms: Fix picking the database from the LB domain

https://gerrit.wikimedia.org/r/1031177

Change #1031410 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Fix picking the database from the LB domain

https://gerrit.wikimedia.org/r/1031410

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.6; 2024-05-21).May 14 2024, 11:00 AM

Change #1031177 merged by jenkins-bot:

[mediawiki/core@wmf/1.43.0-wmf.5] rdbms: Fix picking the database from the LB domain

https://gerrit.wikimedia.org/r/1031177

Mentioned in SAL (#wikimedia-operations) [2024-05-14T11:02:53Z] <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:1031177|rdbms: Fix picking the database from the LB domain (T364827)]]

Did this also cause the following?

https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2024.05.14?id=eS5jdY8Bzo4wKbLA1tH8
https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2024.05.14?id=PC5jdY8Bzo4wKbLA2NIC
https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2024.05.14?id=cZlidY8BWp6-Qp2uS6vX

Mentioned in SAL (#wikimedia-operations) [2024-05-14T11:05:31Z] <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:1031177|rdbms: Fix picking the database from the LB domain (T364827)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

if you used virtual domains, yes.

Yes it does use virtual domains.

Mentioned in SAL (#wikimedia-operations) [2024-05-14T11:18:40Z] <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:1031177|rdbms: Fix picking the database from the LB domain (T364827)]] (duration: 15m 47s)

Ladsgroup closed this task as Resolved.May 14 2024, 11:19 AM

Maintenance_bot moved this task from In progress to Done on the DBA board.May 14 2024, 11:29 AM

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.5; 2024-05-14); removed MW-1.43-notes (1.43.0-wmf.6; 2024-05-21).May 14 2024, 12:00 PM

In T364827#9793877, @Ladsgroup wrote:

That code is used in other places now, making the roll back more complicated. The fix is straightforward.

Cool, that makes sense. I note you have proposed https://gerrit.wikimedia.org/r/1031414 rdbms: Regression tests for T364827 which fulfill what I had in mind. Thank you for the quick fix!

Change #1031414 merged by jenkins-bot: