Page MenuHomePhabricator

New term store connects to the wrong host in clients
Closed, ResolvedPublicBUG REPORT

Description

Caused an outage yesterday.

Event Timeline

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptAug 8 2019, 11:38 AM
Ladsgroup changed the subtype of this task from "Task" to "Bug Report".Aug 8 2019, 12:04 PM

Error logs:

12019-08-07 18:07:45 [XUsTcQpAMFMAAK16FvgAAACR] mw1248 cawiki 1.34.0-wmf.16 DBConnection ERROR: Error connecting to 10.64.48.151 as user wikiuser: Unknown database 'cawiki' {"db_server":"10.64.48.151","db_name":"cawiki","db_user":"<>","error":"Unknown database 'cawiki'","trace":"#0 /srv/mediawiki/php-1.34.0-wmf.16/includes/libs/rdbms/database/DatabaseMysqlBase.php(145): Wikimedia\\Rdbms\\Database->newExceptionAfterConnectError()
2#1 /srv/mediawiki/php-1.34.0-wmf.16/includes/libs/rdbms/database/Database.php(309): Wikimedia\\Rdbms\\DatabaseMysqlBase->open()
3#2 /srv/mediawiki/php-1.34.0-wmf.16/includes/libs/rdbms/database/Database.php(292): Wikimedia\\Rdbms\\Database->doInitConnection()
4#3 /srv/mediawiki/php-1.34.0-wmf.16/includes/libs/rdbms/database/Database.php(415): Wikimedia\\Rdbms\\Database->initConnection()
5#4 /srv/mediawiki/php-1.34.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1304): Wikimedia\\Rdbms\\Database::factory()
6#5 /srv/mediawiki/php-1.34.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1088): Wikimedia\\Rdbms\\LoadBalancer->reallyOpenConnection()
7#6 /srv/mediawiki/php-1.34.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(910): Wikimedia\\Rdbms\\LoadBalancer->getLocalConnection()
8#7 /srv/mediawiki/php-1.34.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(886): Wikimedia\\Rdbms\\LoadBalancer->getServerConnection()
9#8 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermIdsResolver.php(229): Wikimedia\\Rdbms\\LoadBalancer->getConnection()
10#9 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermIdsResolver.php(149): Wikibase\\Lib\\Store\\Sql\\Terms\\DatabaseTermIdsResolver->getDbr()
11#10 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Wikibase/lib/includes/Store/Sql/Terms/CachedDatabasePropertyLabelResolver.php(54): Wikibase\\Lib\\Store\\Sql\\Terms\\DatabaseTermIdsResolver->resolveTermsViaJoin()
12#11 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Wikibase/lib/includes/Store/AbstractTermPropertyLabelResolver.php(106): Wikibase\\Lib\\Store\\Sql\\Terms\\CachedDatabasePropertyLabelResolver->loadProperties()
13#12 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Wikibase/lib/includes/Store/AbstractTermPropertyLabelResolver.php(77): Wikibase\\Lib\\Store\\AbstractTermPropertyLabelResolver->getLabelMap()
14#13 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Wikibase/client/includes/DataAccess/PropertyIdResolver.php(81): Wikibase\\Lib\\Store\\AbstractTermPropertyLabelResolver->getPropertyIdsForLabels()
15#14 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Wikibase/client/includes/DataAccess/PropertyIdResolver.php(59): Wikibase\\Client\\DataAccess\\PropertyIdResolver->findPropertyByLabel()
16#15 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Wikibase/client/includes/DataAccess/Scribunto/Scribunto_LuaWikibaseLibrary.php(734): Wikibase\\Client\\DataAccess\\PropertyIdResolver->resolvePropertyId()
17#16 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(391): Wikibase\\Client\\DataAccess\\Scribunto\\Scribunto_LuaWikibaseLibrary->resolvePropertyId()
18#17 (): Scribunto_LuaSandboxCallback->__call()
19#18 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(314): LuaSandboxFunction->call()
20#19 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(296): Scribunto_LuaSandboxInterpreter->callFunction()
21#20 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(983): Scribunto_LuaEngine->executeFunctionChunk()
22#21 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/common/Hooks.php(128): Scribunto_LuaModule->invoke()
23#22 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3607): ScribuntoHooks::invokeHook()
24#23 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3312): Parser->callParserFunction()
25#24 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
26#25 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3229): PPFrame_Hash->expand()
27#26 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
28#27 /srv/mediawiki/php-1.34.0-wmf.16/extensions/ParserFunctions/includes/ParserFunctions.php(123): PPFrame_Hash->expand()
29#28 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3607): MediaWiki\\Extensions\\ParserFunctions\\ParserFunctions::if()
30#29 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3312): Parser->callParserFunction()
31#30 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
32#31 /srv/mediawiki/php-1.34.0-wmf.16/extensions/ParserFunctions/includes/ParserFunctions.php(123): PPFrame_Hash->expand()
33#32 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3607): MediaWiki\\Extensions\\ParserFunctions\\ParserFunctions::if()
34#33 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3312): Parser->callParserFunction()
35#34 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
36#35 /srv/mediawiki/php-1.34.0-wmf.16/extensions/ParserFunctions/includes/ParserFunctions.php(123): PPFrame_Hash->expand()
37#36 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3607): MediaWiki\\Extensions\\ParserFunctions\\ParserFunctions::if()
38#37 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3312): Parser->callParserFunction()
39#38 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
40#39 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3488): PPFrame_Hash->expand()
41#40 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
42#41 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPTemplateFrame_Hash.php(167): PPFrame_Hash->expand()
43#42 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPTemplateFrame_Hash.php(179): PPTemplateFrame_Hash->getNamedArgument()
44#43 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(676): PPTemplateFrame_Hash->getArgument()
45#44 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(391): Scribunto_LuaEngine->getExpandedArgument()
46#45 (): Scribunto_LuaSandboxCallback->__call()
47#46 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(314): LuaSandboxFunction->call()
48#47 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(296): Scribunto_LuaSandboxInterpreter->callFunction()
49#48 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(983): Scribunto_LuaEngine->executeFunctionChunk()
50#49 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/common/Hooks.php(128): Scribunto_LuaModule->invoke()
51#50 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3607): ScribuntoHooks::invokeHook()
52#51 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3312): Parser->callParserFunction()
53#52 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
54#53 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPTemplateFrame_Hash.php(167): PPFrame_Hash->expand()
55#54 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPTemplateFrame_Hash.php(179): PPTemplateFrame_Hash->getNamedArgument()
56#55 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3990): PPTemplateFrame_Hash->getArgument()
57#56 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(271): Parser->argSubstitution()
58#57 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3229): PPFrame_Hash->expand()
59#58 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
60#59 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPTemplateFrame_Hash.php(167): PPFrame_Hash->expand()
61#60 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPTemplateFrame_Hash.php(179): PPTemplateFrame_Hash->getNamedArgument()
62#61 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(676): PPTemplateFrame_Hash->getArgument()
63#62 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(391): Scribunto_LuaEngine->getExpandedArgument()
64#63 (): Scribunto_LuaSandboxCallback->__call()
65#64 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaSandbox/Engine.php(314): LuaSandboxFunction->call()
66#65 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(296): Scribunto_LuaSandboxInterpreter->callFunction()
67#66 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/engines/LuaCommon/LuaCommon.php(983): Scribunto_LuaEngine->executeFunctionChunk()
68#67 /srv/mediawiki/php-1.34.0-wmf.16/extensions/Scribunto/includes/common/Hooks.php(128): Scribunto_LuaModule->invoke()
69#68 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3607): ScribuntoHooks::invokeHook()
70#69 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3312): Parser->callParserFunction()
71#70 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
72#71 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3488): PPFrame_Hash->expand()
73#72 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
74#73 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3488): PPFrame_Hash->expand()
75#74 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
76#75 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPTemplateFrame_Hash.php(87): PPFrame_Hash->expand()
77#76 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3485): PPTemplateFrame_Hash->cachedExpand()
78#77 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/PPFrame_Hash.php(254): Parser->braceSubstitution()
79#78 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(3126): PPFrame_Hash->expand()
80#79 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(1437): Parser->replaceVariables()
81#80 /srv/mediawiki/php-1.34.0-wmf.16/includes/parser/Parser.php(568): Parser->internalParse()
82#81 /srv/mediawiki/php-1.34.0-wmf.16/includes/content/WikitextContent.php(365): Parser->parse()
83#82 /srv/mediawiki/php-1.34.0-wmf.16/includes/content/AbstractContent.php(555): WikitextContent->fillParserOutput()
84#83 /srv/mediawiki/php-1.34.0-wmf.16/includes/Revision/RenderedRevision.php(266): AbstractContent->getParserOutput()
85#84 /srv/mediawiki/php-1.34.0-wmf.16/includes/Revision/RenderedRevision.php(234): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutputUncached()
86#85 /srv/mediawiki/php-1.34.0-wmf.16/includes/Revision/RevisionRenderer.php(222): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutput()
87#86 /srv/mediawiki/php-1.34.0-wmf.16/includes/Revision/RevisionRenderer.php(151): MediaWiki\\Revision\\RevisionRenderer->combineSlotOutput()
88#87 (): Closure$MediaWiki\\Revision\\RevisionRenderer::getRenderedRevision#3()
89#88 /srv/mediawiki/php-1.34.0-wmf.16/includes/Revision/RenderedRevision.php(197): call_user_func()
90#89 /srv/mediawiki/php-1.34.0-wmf.16/includes/poolcounter/PoolWorkArticleView.php(196): MediaWiki\\Revision\\RenderedRevision->getRevisionParserOutput()
91#90 /srv/mediawiki/php-1.34.0-wmf.16/includes/poolcounter/PoolCounterWork.php(125): PoolWorkArticleView->doWork()
92#91 /srv/mediawiki/php-1.34.0-wmf.16/includes/page/Article.php(776): PoolCounterWork->execute()
93#92 /srv/mediawiki/php-1.34.0-wmf.16/includes/actions/ViewAction.php(63): Article->view()
94#93 /srv/mediawiki/php-1.34.0-wmf.16/includes/MediaWiki.php(507): ViewAction->show()
95#94 /srv/mediawiki/php-1.34.0-wmf.16/includes/MediaWiki.php(302): MediaWiki->performAction()
96#95 /srv/mediawiki/php-1.34.0-wmf.16/includes/MediaWiki.php(892): MediaWiki->performRequest()
97#96 /srv/mediawiki/php-1.34.0-wmf.16/includes/MediaWiki.php(523): MediaWiki->main()
98#97 /srv/mediawiki/php-1.34.0-wmf.16/index.php(42): MediaWiki->run()
99#98 /srv/mediawiki/w/index.php(3): include()
100#99 {main}"}

Change 529113 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Fix DB domain for DatabaseTermIdsResolver in client

https://gerrit.wikimedia.org/r/529113

Change 529113 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Fix DB domain for DatabaseTermIdsResolver in client

https://gerrit.wikimedia.org/r/529113

Change 529742 had a related patch set uploaded (by Alaa Sarhan; owner: Alaa Sarhan):
[mediawiki/extensions/Wikibase@master] Initialize DatabaseTypeIdsStore with repo database name in client

https://gerrit.wikimedia.org/r/529742

Another database access needed to be updated the same way

Change 529742 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Initialize DatabaseTypeIdsStore with repo database name in client

https://gerrit.wikimedia.org/r/529742

Change 530841 had a related patch set uploaded (by Urbanecm; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.34.0-wmf.17] Fix DB domain for DatabaseTermIdsResolver in client

https://gerrit.wikimedia.org/r/530841

Change 530842 had a related patch set uploaded (by Urbanecm; owner: Alaa Sarhan):
[mediawiki/extensions/Wikibase@wmf/1.34.0-wmf.17] Initialize DatabaseTypeIdsStore with repo database name in client

https://gerrit.wikimedia.org/r/530842

Change 530845 had a related patch set uploaded (by Alaa Sarhan; owner: Alaa Sarhan):
[mediawiki/extensions/Wikibase@wmf/1.34.0-wmf.17] Initialize DatabaseTermIdsResolver and DatabaseTypeIdsStore with repo database name in client.

https://gerrit.wikimedia.org/r/530845

Change 530842 abandoned by Alaa Sarhan:
Initialize DatabaseTypeIdsStore with repo database name in client

Reason:
For https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/ /530845

https://gerrit.wikimedia.org/r/530842

Change 530841 abandoned by Alaa Sarhan:
Fix DB domain for DatabaseTermIdsResolver in client

Reason:
For https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/ /530845

https://gerrit.wikimedia.org/r/530841

Change 530845 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.34.0-wmf.17] Initialize DatabaseTermIdsResolver and DatabaseTypeIdsStore with repo database name in client.

https://gerrit.wikimedia.org/r/530845

Mentioned in SAL (#wikimedia-operations) [2019-08-20T12:05:55Z] <awight@deploy1001> Synchronized php-1.34.0-wmf.17/extensions/Wikibase: SWAT: [[gerrit:530845|Initialize DatabaseTermIdsResolver and DatabaseTypeIdsStore with repo database name in client. (T230119, T225053)]] (duration: 00m 52s)