Page MenuHomePhabricator

Wikimedia\Rdbms\DBQueryError: Error 1049: Unknown database 'cognate_wiktionary'
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1049: Unknown database 'cognate_wiktionary'
Function: Wikimedia\Rdbms\DatabaseMySQL::doSelectDomain
Query: USE `cognate_wiktionary`
exception.trace
from /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/Database.php(1203)
#0 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/Database.php(1187): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/Database.php(1161): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/DatabaseMySQL.php(204): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string)
#3 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/Database.php(1514): Wikimedia\Rdbms\DatabaseMySQL->doSelectDomain(Wikimedia\Rdbms\DatabaseDomain)
#4 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/loadbalancer/LoadBalancer.php(912): Wikimedia\Rdbms\Database->selectDomain(Wikimedia\Rdbms\DatabaseDomain)
#5 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/loadbalancer/LoadBalancer.php(801): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(integer, Wikimedia\Rdbms\DatabaseDomain, integer)
#6 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/loadbalancer/LoadBalancer.php(793): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#7 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/DBConnRef.php(110): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#8 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/DBConnRef.php(124): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#9 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/database/DBConnRef.php(358): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#10 /srv/mediawiki/php-1.43.0-wmf.5/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(730): Wikimedia\Rdbms\DBConnRef->select(array, array, array, string, array, array)
#11 /srv/mediawiki/php-1.43.0-wmf.5/extensions/Cognate/src/CognateStore.php(175): Wikimedia\Rdbms\SelectQueryBuilder->fetchResultSet()
#12 /srv/mediawiki/php-1.43.0-wmf.5/extensions/Cognate/src/CognateRepo.php(140): Cognate\CognateStore->selectLinkDetailsForPage(string, MediaWiki\Title\Title)
#13 /srv/mediawiki/php-1.43.0-wmf.5/extensions/Cognate/src/HookHandler/CognateParseHookHandler.php(79): Cognate\CognateRepo->getLinksForPage(string, MediaWiki\Title\Title)
#14 /srv/mediawiki/php-1.43.0-wmf.5/extensions/Cognate/src/CognateHooks.php(125): Cognate\HookHandler\CognateParseHookHandler->doContentAlterParserOutput(MediaWiki\Title\Title, MediaWiki\Parser\ParserOutput)
#15 /srv/mediawiki/php-1.43.0-wmf.5/includes/HookContainer/HookContainer.php(159): Cognate\CognateHooks->onContentAlterParserOutput(WikitextContent, MediaWiki\Title\Title, MediaWiki\Parser\ParserOutput)
#16 /srv/mediawiki/php-1.43.0-wmf.5/includes/HookContainer/HookRunner.php(1185): MediaWiki\HookContainer\HookContainer->run(string, array)
#17 /srv/mediawiki/php-1.43.0-wmf.5/includes/content/ContentHandler.php(1690): MediaWiki\HookContainer\HookRunner->onContentAlterParserOutput(WikitextContent, MediaWiki\Title\Title, MediaWiki\Parser\ParserOutput)
#18 /srv/mediawiki/php-1.43.0-wmf.5/includes/content/Renderer/ContentRenderer.php(67): ContentHandler->getParserOutput(WikitextContent, MediaWiki\Content\Renderer\ContentParseParams)
#19 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiParse.php(161): MediaWiki\Content\Renderer\ContentRenderer->getParserOutput(WikitextContent, MediaWiki\Title\Title, NULL, ParserOptions)
#20 /srv/mediawiki/php-1.43.0-wmf.5/includes/poolcounter/PoolCounterWorkViaCallback.php(73): ApiParse->{closure}()
#21 /srv/mediawiki/php-1.43.0-wmf.5/includes/poolcounter/PoolCounterWork.php(171): MediaWiki\PoolCounter\PoolCounterWorkViaCallback->doWork()
#22 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiParse.php(169): MediaWiki\PoolCounter\PoolCounterWork->execute()
#23 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiParse.php(446): ApiParse->getContentParserOutput(WikitextContent, MediaWiki\Title\Title, NULL, ParserOptions)
#24 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiMain.php(1952): ApiParse->execute()
#25 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiMain.php(928): ApiMain->executeAction()
#26 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiMain.php(899): ApiMain->executeActionWithErrorHandling()
#27 /srv/mediawiki/php-1.43.0-wmf.5/includes/api/ApiEntryPoint.php(158): ApiMain->execute()
#28 /srv/mediawiki/php-1.43.0-wmf.5/includes/MediaWikiEntryPoint.php(199): MediaWiki\Api\ApiEntryPoint->execute()
#29 /srv/mediawiki/php-1.43.0-wmf.5/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#30 /srv/mediawiki/w/api.php(3): require(string)
#31 {main}
Impact

Logspam, and user scripts querying the aawiktionary API don’t work.

Notes

aawiktionary is a closed wiki, I don’t know if that’s relevant.
As far as I understand, cognate_wiktionary is a database that’s shared across several wikis.

Details

Event Timeline

From that error, db1175 belongs to s3, which doesn't store cognate_wiktionary - that database lives in x1.

aawiktionary is a closed wiki, I don’t know if that’s relevant.

Doesn’t seem like it, I can also reproduce it on dewiktionary as long as the &title= doesn’t exist. (aawiktionary then just has the special property that virtually every title is guaranteed to not exist.) edit: wrong, see below

Possibly related to T348526: [COG] [TECH] Migrate Cognate to use a virtual database domain? But given T364828, it seems more likely that something’s generally broken about shared (x1) databases.

hashar triaged this task as Unbreak Now! priority.Tue, May 14, 9:44 AM

This sounds like it’s not a train issue, by the way, since dewiktionary (group1) is still on wmf.4… unless it’s somehow loading wmf.5 code anyway? (The timing does feel suspicious, I’ll admit that.) edit: wrong, see below

This sounds like it’s not a train issue, by the way, since dewiktionary (group1) is still on wmf.4… unless it’s somehow loading wmf.5 code anyway? (The timing does feel suspicious, I’ll admit that.)

Scratch that, I somehow failed to actually change the URL to dewiktionary. Cannot reproduce the issue on dewiktionary after all, so it probably is the train.

I think I know what's going on. Give me a bit.

This is very likely caused by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1024973/3/includes/libs/rdbms/lbfactory/LBFactory.php a refactor we did on virtual domains, but I can't see how. I'm not spotting any obvious mistakes (I dig deeper). If you do, please let me know.

Ladsgroup added a project: DBA.

Ah found it. Sneaky bug. I patch it now.

Change #1031410 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Fix picking the database from the LB domain

https://gerrit.wikimedia.org/r/1031410

@Ladsgroup can you consider rolling back the code from MediaWiki and repurpose it for next week with test coverage added? I'd like to avoid monkey patching code :-)

That code is used in other places now, making the roll back more complicated. The fix is straightforward.

Change #1031414 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Regression tests for T364827

https://gerrit.wikimedia.org/r/1031414

Change #1031177 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.43.0-wmf.5] rdbms: Fix picking the database from the LB domain

https://gerrit.wikimedia.org/r/1031177

Change #1031410 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Fix picking the database from the LB domain

https://gerrit.wikimedia.org/r/1031410

Change #1031177 merged by jenkins-bot:

[mediawiki/core@wmf/1.43.0-wmf.5] rdbms: Fix picking the database from the LB domain

https://gerrit.wikimedia.org/r/1031177

Mentioned in SAL (#wikimedia-operations) [2024-05-14T11:02:53Z] <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:1031177|rdbms: Fix picking the database from the LB domain (T364827)]]

Mentioned in SAL (#wikimedia-operations) [2024-05-14T11:05:31Z] <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:1031177|rdbms: Fix picking the database from the LB domain (T364827)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

if you used virtual domains, yes.

Mentioned in SAL (#wikimedia-operations) [2024-05-14T11:18:40Z] <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:1031177|rdbms: Fix picking the database from the LB domain (T364827)]] (duration: 15m 47s)

That code is used in other places now, making the roll back more complicated. The fix is straightforward.

Cool, that makes sense. I note you have proposed https://gerrit.wikimedia.org/r/1031414 rdbms: Regression tests for T364827 which fulfill what I had in mind. Thank you for the quick fix!

Change #1031414 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Regression tests for T364827

https://gerrit.wikimedia.org/r/1031414