Page MenuHomePhabricator

Foreign query for metawiki fails with "Table 'centralauth.page' doesn't exist" (DBConnRef mixup?)
Open, HighPublicPRODUCTION ERROR

Description

Spotted in the logs. It seems like getCentralTouched() gets a handle for the wrong database (e.g. selectDB). The method itself looks fine, so I'm not sure how that is possible. That should never happen.

A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SELECT  page_touched,pp_propname  FROM `page` LEFT JOIN `page_props` ON ((page_id=pp_page) AND pp_propname = 'noglobal')   WHERE page_namespace = '2' AND page_title = 'Bujo'  LIMIT 1  
Function: MediaWiki\GlobalUserPage\GlobalUserPage::getCentralTouched
Error: 1146 Table 'centralauth.page' doesn't exist (10.64.0.91)

#0 /srv/mediawiki/php-1.31.0-wmf.30/includes/libs/rdbms/database/Database.php(1422): Wikimedia\Rdbms\Database->makeQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.31.0-wmf.30/includes/libs/rdbms/database/Database.php(1195): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /srv/mediawiki/php-1.31.0-wmf.30/includes/libs/rdbms/database/Database.php(1648): Wikimedia\Rdbms\Database->query(string, string)
#3 /srv/mediawiki/php-1.31.0-wmf.30/includes/libs/rdbms/database/Database.php(1725): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#4 [internal function]: Wikimedia\Rdbms\Database->selectRow(array, array, array, string, array, array)
#5 /srv/mediawiki/php-1.31.0-wmf.30/includes/libs/rdbms/database/DBConnRef.php(49): call_user_func_array(array, array)
#6 /srv/mediawiki/php-1.31.0-wmf.30/includes/libs/rdbms/database/DBConnRef.php(268): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.31.0-wmf.30/extensions/GlobalUserPage/includes/GlobalUserPage.php(212): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#8 /srv/mediawiki/php-1.31.0-wmf.30/extensions/GlobalUserPage/includes/GlobalUserPage.php(174): MediaWiki\GlobalUserPage\GlobalUserPage::getCentralTouched(User)
#9 /srv/mediawiki/php-1.31.0-wmf.30/extensions/GlobalUserPage/includes/Hooks.php(57): MediaWiki\GlobalUserPage\GlobalUserPage::shouldDisplayGlobalPage(Title)
#10 /srv/mediawiki/php-1.31.0-wmf.30/includes/Hooks.php(177): MediaWiki\GlobalUserPage\Hooks::onTitleIsAlwaysKnown(Title, NULL)
#11 /srv/mediawiki/php-1.31.0-wmf.30/includes/Hooks.php(205): Hooks::callHook(string, array, array, NULL)
#12 /srv/mediawiki/php-1.31.0-wmf.30/includes/Title.php(4589): Hooks::run(string, array)
#13 /srv/mediawiki/php-1.31.0-wmf.30/includes/Title.php(4630): Title->isAlwaysKnown()
#14 /srv/mediawiki/php-1.31.0-wmf.30/includes/skins/Skin.php(1200): Title->isKnown()
#15 /srv/mediawiki/php-1.31.0-wmf.30/includes/skins/SkinTemplate.php(220): Skin::makeUrlDetails(string)
#16 /srv/mediawiki/php-1.31.0-wmf.30/includes/skins/SkinTemplate.php(303): SkinTemplate->setupTemplateForOutput()
#17 /srv/mediawiki/php-1.31.0-wmf.30/includes/skins/SkinTemplate.php(249): SkinTemplate->prepareQuickTemplate()
#18 /srv/mediawiki/php-1.31.0-wmf.30/includes/OutputPage.php(2388): SkinTemplate->outputPage()
#19 /srv/mediawiki/php-1.31.0-wmf.30/includes/MediaWiki.php(868): OutputPage->output(boolean)
#20 /srv/mediawiki/php-1.31.0-wmf.30/includes/libs/rdbms/lbfactory/LBFactory.php(496): Closure$MediaWiki::main()
#21 /srv/mediawiki/php-1.31.0-wmf.30/includes/libs/rdbms/lbfactory/LBFactory.php(153): Wikimedia\Rdbms\LBFactory->shutdownChronologyProtector(Wikimedia\Rdbms\ChronologyProtector, Closure$MediaWiki::main;1350, string, integer)
#22 /srv/mediawiki/php-1.31.0-wmf.30/includes/MediaWiki.php(634): Wikimedia\Rdbms\LBFactory->shutdown(integer, Closure$MediaWiki::main;1350, integer)
#23 /srv/mediawiki/php-1.31.0-wmf.30/includes/MediaWiki.php(575): MediaWiki::preOutputCommit(RequestContext, Closure$MediaWiki::main;1350)
#24 /srv/mediawiki/php-1.31.0-wmf.30/includes/MediaWiki.php(877): MediaWiki->doPreOutputCommit(Closure$MediaWiki::main;1350)
#25 /srv/mediawiki/php-1.31.0-wmf.30/includes/MediaWiki.php(524): MediaWiki->main()
#26 /srv/mediawiki/php-1.31.0-wmf.30/index.php(42): MediaWiki->run()
#27 /srv/mediawiki/w/index.php(3): include(string)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I tried importing a file into testwiki with curl, forcing a centralauth DB connection in the same request by first deleting the global:centralauth-user:... cache key, still could not reproduce.

It doesn't have to be a LoadBalancer bug, it could just be some other extension calling reuseConnection() inappropriately. It's hard to debug without a reproduction procedure. I see in the logs that there was a series of these on 2018-08-06 with URL https://sat.wikipedia.org/w/index.php?title=%E1%B1%9F%E1%B1%A5%E1%B1%9A%E1%B1%A0%E1%B1%9F%E1%B1%AD:Import&action=submit , and the failed query indicates that the user was @MF-Warburg , who did have successful file upload imports at that time in the logs: https://sat.wikipedia.org/wiki/%E1%B1%9F%E1%B1%A5%E1%B1%9A%E1%B1%A0%E1%B1%9F%E1%B1%AD:Log/import

I wasn't able to reproduce it by doing interwiki imports into testwiki, but maybe I wasn't hitting CentralAuth in the same way.

@MF-Warburg, to help us isolate this bug, can you tell us what kind of client you were using? If you used a browser, did you see errors when you did some of these imports, and was there anything special about the ones that failed?

I used Firefox 61.0.2. There were some errors, but I don't remember what exactly they said - but I see them frequently when importing: it's an error message that is shown on a page different from the normal wiki layout. Sometimes they cause pages not to be imported at all, and sometimes all pages to be imported from the XML file actually do get imported, but an error message is still shown.
Today I was notified that some pages still need to be imported to satwiki (maybe I missed them last time because of this error precisely), so if errors occur again, I can tell you exactly what the error message is.

Change 452878 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] [WIP] Database::selectDB() update the domain field and handle failure better

https://gerrit.wikimedia.org/r/452878

Change 452885 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: Add domain sanity check to LoadBalancer::openForeignConnection

https://gerrit.wikimedia.org/r/452885

Change 452616 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: Disable DBConnRef::selectDB() for sanity

https://gerrit.wikimedia.org/r/452616

Change 452616 merged by jenkins-bot:
[mediawiki/core@master] rdbms: Disable DBConnRef::selectDB() for sanity

https://gerrit.wikimedia.org/r/452616

Today I was notified that some pages still need to be imported to satwiki (maybe I missed them last time because of this error precisely), so if errors occur again, I can tell you exactly what the error message is.

For the record, today when importing to satwiki there were no errors.

Found the following in the logs, which suggest this might be happening from another code path as well..

Request ID: W7PYJgrAADQAAJscsOMAAAAH

message
Wikimedia\Rdbms\DBConnectionError:
Cannot access the database: Error selecting database 'centralauth' on server ##.##.##.##
trace
#0 /srv/mediawiki/php-1.32.0-wmf.23/includes/libs/rdbms/loadbalancer/LoadBalancer.php(768): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-1.32.0-wmf.23/includes/libs/rdbms/loadbalancer/LoadBalancer.php(850): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#2 /srv/mediawiki/php-1.32.0-wmf.23/extensions/CentralAuth/includes/CentralAuthUtils.php(70): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, string, string)
#3 /srv/mediawiki/php-1.32.0-wmf.23/extensions/CentralAuth/includes/CentralAuthUser.php(225): CentralAuthUtils::getCentralSlaveDB()
#4 /srv/mediawiki/php-1.32.0-wmf.23/extensions/CentralAuth/includes/CentralAuthUser.php(2241): CentralAuthUser->getSafeReadDB()
#5 /srv/mediawiki/php-1.32.0-wmf.23/extensions/CentralAuth/includes/CentralAuthUser.php(538): CentralAuthUser->loadAttached()
#6 /srv/mediawiki/php-1.32.0-wmf.23/extensions/CentralAuth/includes/CentralAuthUser.php(521): CentralAuthUser->loadFromCacheObject(array)
#7 /srv/mediawiki/php-1.32.0-wmf.23/extensions/CentralAuth/includes/CentralAuthUser.php(374): CentralAuthUser->loadFromCache()
#8 /srv/mediawiki/php-1.32.0-wmf.23/extensions/CentralAuth/includes/CentralAuthUser.php(597): CentralAuthUser->loadState()
#9 /srv/mediawiki/php-1.32.0-wmf.23/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php(262): CentralAuthUser->isAttached()
#10 /srv/mediawiki/php-1.32.0-wmf.23/includes/session/CookieSessionProvider.php(199): CentralAuthSessionProvider->sessionDataToExport(User)
#11 /srv/mediawiki/php-1.32.0-wmf.23/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php(282): MediaWiki\Session\CookieSessionProvider->persistSession(MediaWiki\Session\SessionBackend, WebRequest)
#12 /srv/mediawiki/php-1.32.0-wmf.23/includes/session/SessionBackend.php(687): CentralAuthSessionProvider->persistSession(MediaWiki\Session\SessionBackend, WebRequest)
#13 /srv/mediawiki/php-1.32.0-wmf.23/includes/session/SessionBackend.php(607): MediaWiki\Session\SessionBackend->save()
#14 /srv/mediawiki/php-1.32.0-wmf.23/includes/session/SessionBackend.php(581): MediaWiki\Session\SessionBackend->autosave()
#15 /srv/mediawiki/php-1.32.0-wmf.23/includes/session/SessionBackend.php(293): MediaWiki\Session\SessionBackend->renew()
#16 /srv/mediawiki/php-1.32.0-wmf.23/includes/session/Session.php(127): MediaWiki\Session\SessionBackend->persist()
#17 /srv/mediawiki/php-1.32.0-wmf.23/includes/session/PHPSessionHandler.php(330): MediaWiki\Session\Session->persist()
#18 [internal function]: MediaWiki\Session\PHPSessionHandler->write(string, string)
#19 /srv/mediawiki/php-1.32.0-wmf.23/includes/session/SessionManager.php(470): session_write_close()
#20 [internal function]: MediaWiki\Session\SessionManager->shutdown()

The same request ID also reported:

PHP Fatal Error: entire web request took longer than 60 seconds and timed out

#0 /srv/mediawiki/php-1.32.0-wmf.23/includes/diff/TextSlotDiffRenderer.php(236): NO_FUNCTION_GIVEN()
#1 /srv/mediawiki/php-1.32.0-wmf.23/includes/diff/TextSlotDiffRenderer.php(152): TextSlotDiffRenderer::getTextDiffInternal()
#2 /srv/mediawiki/php-1.32.0-wmf.23/includes/poolcounter/PoolCounterWorkViaCallback.php(69): Closure$TextSlotDiffRenderer::getTextDiff()
#3 /srv/mediawiki/php-1.32.0-wmf.23/includes/poolcounter/PoolCounterWork.php(123): PoolCounterWorkViaCallback->doWork()
#4 /srv/mediawiki/php-1.32.0-wmf.23/includes/diff/TextSlotDiffRenderer.php(185): PoolCounterWork->execute()
#5 /srv/mediawiki/php-1.32.0-wmf.23/includes/diff/TextSlotDiffRenderer.php(136): TextSlotDiffRenderer->getTextDiff(string, string)
#6 /srv/mediawiki/php-1.32.0-wmf.23/includes/diff/DifferenceEngine.php(1055): TextSlotDiffRenderer->getDiff(WikitextContent, WikitextContent)
#7 /srv/mediawiki/php-1.32.0-wmf.23/includes/diff/DifferenceEngine.php(973): DifferenceEngine->getDiffBody()
#8 /srv/mediawiki/php-1.32.0-wmf.23/includes/diff/DifferenceEngine.php(938): DifferenceEngine->getDiff(string, string, string)
#9 /srv/mediawiki/php-1.32.0-wmf.23/includes/diff/DifferenceEngine.php(706): DifferenceEngine->showDiff(string, string, string)
#10 /srv/mediawiki/php-1.32.0-wmf.23/includes/page/Article.php(930): DifferenceEngine->showDiffPage(boolean)
#11 /srv/mediawiki/php-1.32.0-wmf.23/includes/page/Article.php(615): Article->showDiffPage()
#12 /srv/mediawiki/php-1.32.0-wmf.23/includes/actions/ViewAction.php(68): Article->view()

I think this may relate to T204346. It appears that, when HHVM aborts a web request that is taking too long, something strange happens to its database connections.

MediaWIki's shutdown handlers aren't aware of that state change, and do all kinds of bad things. Including, sending invalid COMMIT commands to the database (T204346), and sending queries to the wrong database server (this task).

Change 452878 merged by jenkins-bot:
[mediawiki/core@master] rdbms: Database::selectDB() update the domain and handle failure better

https://gerrit.wikimedia.org/r/452878

Change 452885 merged by Krinkle:
[mediawiki/core@master] rdbms: add domain sanity checks to LoadBalancer connection methods

https://gerrit.wikimedia.org/r/452885

Change 467078 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: re-add DB domain sanity checks to LoadBalancer

https://gerrit.wikimedia.org/r/467078

Change 467844 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@wmf/1.32.0-wmf.26] rdbms: re-add DB domain sanity checks to LoadBalancer

https://gerrit.wikimedia.org/r/467844

Change 467078 merged by jenkins-bot:
[mediawiki/core@master] rdbms: re-add DB domain sanity checks to LoadBalancer

https://gerrit.wikimedia.org/r/467078

Fixed in master.

Change 467874 had a related patch set uploaded (by Jforrester; owner: Aaron Schulz):
[mediawiki/core@REL1_32] rdbms: re-add DB domain sanity checks to LoadBalancer

https://gerrit.wikimedia.org/r/467874

Change 467874 merged by jenkins-bot:
[mediawiki/core@REL1_32] rdbms: re-add DB domain sanity checks to LoadBalancer

https://gerrit.wikimedia.org/r/467874

Change 467844 abandoned by Aaron Schulz:
rdbms: re-add DB domain sanity checks to LoadBalancer

Reason:
obsolete

https://gerrit.wikimedia.org/r/467844

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM
This comment was removed by saper.

Change 571108 had a related patch set uploaded (by saper; owner: Aaron Schulz):
[mediawiki/core@REL1_31] rdbms: re-add DB domain sanity checks to LoadBalancer

https://gerrit.wikimedia.org/r/571108

Change 571108 merged by jenkins-bot:
[mediawiki/core@REL1_31] rdbms: re-add DB domain sanity checks to LoadBalancer

https://gerrit.wikimedia.org/r/571108

Krinkle reopened this task as Open.EditedMay 19 2020, 2:46 AM

Oops, still seen.

  • reqId: 1f0d1ec2-7b02-4522-be0e-e3e68fcc3b71
Table 'metawiki.localuser' doesn't exist (10.64.32.198:3317)
 
severity: ERROR
channel: DBQuery
message:
 CentralAuthUser::loadAttached	10.64.32.198:3317	1146	Table 'metawiki.localuser' doesn't exist (10.64.32.198:3317)	SELECT  lu_wiki  FROM `localuser`    WHERE lu_name = x
trace
#0 /srv/mediawiki/php-1.35.0-wmf.31/includes/libs/rdbms/database/Database.php(1640): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#1 /srv/mediawiki/php-1.35.0-wmf.31/includes/libs/rdbms/database/Database.php(1215): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /srv/mediawiki/php-1.35.0-wmf.31/includes/libs/rdbms/database/Database.php(1879): Wikimedia\Rdbms\Database->query(string, string, integer)
#3 /srv/mediawiki/php-1.35.0-wmf.31/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->select(string, array, array, string)
#4 /srv/mediawiki/php-1.35.0-wmf.31/includes/libs/rdbms/database/DBConnRef.php(313): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#5 /srv/mediawiki/php-1.35.0-wmf.31/extensions/CentralAuth/includes/CentralAuthUser.php(2286): Wikimedia\Rdbms\DBConnRef->select(string, array, array, string)
#6 /srv/mediawiki/php-1.35.0-wmf.31/extensions/CentralAuth/includes/CentralAuthUser.php(578): CentralAuthUser->loadAttached()
#7 /srv/mediawiki/php-1.35.0-wmf.31/extensions/CentralAuth/includes/CentralAuthUser.php(561): CentralAuthUser->loadFromCacheObject(array)
#8 /srv/mediawiki/php-1.35.0-wmf.31/extensions/CentralAuth/includes/CentralAuthUser.php(414): CentralAuthUser->loadFromCache()
#9 /srv/mediawiki/php-1.35.0-wmf.31/extensions/CentralAuth/includes/CentralAuthUser.php(589): CentralAuthUser->loadState()
#10 /srv/mediawiki/php-1.35.0-wmf.31/extensions/CentralAuth/includes/CentralAuthUser.php(686): CentralAuthUser->getId()
#11 /srv/mediawiki/php-1.35.0-wmf.31/extensions/CentralAuth/includes/CentralAuthHooks.php(925): CentralAuthUser->exists()
#12 /srv/mediawiki/php-1.35.0-wmf.31/includes/HookContainer/HookContainer.php(289): CentralAuthHooks::onGetUserBlock(User, string, MediaWiki\Block\SystemBlock)
#13 /srv/mediawiki/php-1.35.0-wmf.31/includes/HookContainer/HookContainer.php(129): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#14 /srv/mediawiki/php-1.35.0-wmf.31/includes/Hooks.php(136): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#15 /srv/mediawiki/php-1.35.0-wmf.31/includes/block/BlockManager.php(162): Hooks::run(string, array)
#16 /srv/mediawiki/php-1.35.0-wmf.31/includes/user/User.php(1661): MediaWiki\Block\BlockManager->getUserBlock(User, WebRequest, boolean)
#17 /srv/mediawiki/php-1.35.0-wmf.31/includes/user/User.php(1980): User->getBlockedStatus(boolean)
#18 /srv/mediawiki/php-1.35.0-wmf.31/includes/block/BlockManager.php(465): User->getBlock()
#19 /srv/mediawiki/php-1.35.0-wmf.31/includes/MediaWiki.php(757): MediaWiki\Block\BlockManager->trackBlockWithCookie(User, WebResponse)
#20 /srv/mediawiki/php-1.35.0-wmf.31/includes/api/ApiMain.php(549): MediaWiki::preOutputCommit(DerivativeContext)
#21 /srv/mediawiki/php-1.35.0-wmf.31/includes/api/ApiMain.php(494): ApiMain->executeActionWithErrorHandling()
#22 /srv/mediawiki/php-1.35.0-wmf.31/api.php(84): ApiMain->execute()
#23 /srv/mediawiki/w/api.php(3): require(string)
#24 {main}

The request in question was an API request which seems to have fatalled with a DBQueryError exception as a result (given trackBlockWithCookie is called pre-send).

BPirkle removed a project: Platform Engineering.
BPirkle added a subscriber: BPirkle.

Retag CPT when you need code review

Hi @aaron, It looks like this ticket has resurfaced. Should this be a Medium or High Priority? Would you like to work on it when you come over to Clinic Duty around 6/10?

Krinkle triaged this task as High priority.Jun 23 2020, 6:24 PM
Krinkle removed a project: Performance-Team (Radar).

Production error.

Still happening.

DBQueryError:
Error 1146: Table 'metawiki.localuser' doesn't exist (10.64.0.91)
Function: CentralAuthUser::loadAttached
Query: SELECT  lu_wiki  FROM `localuser`    WHERE lu_name = xxx.xx.xx.xx
trace
#0 /srv/mediawiki/php-1.36.0-wmf.3/includes/libs/rdbms/database/Database.php(1683): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.3/includes/libs/rdbms/database/Database.php(1658): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.36.0-wmf.3/includes/libs/rdbms/database/Database.php(1227): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.36.0-wmf.3/includes/libs/rdbms/database/Database.php(1907): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.3/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->select(string, array, array, string)
#5 /srv/mediawiki/php-1.36.0-wmf.3/includes/libs/rdbms/database/DBConnRef.php(313): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.36.0-wmf.3/extensions/CentralAuth/includes/CentralAuthUser.php(2273): Wikimedia\Rdbms\DBConnRef->select(string, array, array, string)
#7 /srv/mediawiki/php-1.36.0-wmf.3/extensions/CentralAuth/includes/CentralAuthUser.php(565): CentralAuthUser->loadAttached()
#8 /srv/mediawiki/php-1.36.0-wmf.3/extensions/CentralAuth/includes/CentralAuthUser.php(548): CentralAuthUser->loadFromCacheObject(array)
#9 /srv/mediawiki/php-1.36.0-wmf.3/extensions/CentralAuth/includes/CentralAuthUser.php(401): CentralAuthUser->loadFromCache()
#10 /srv/mediawiki/php-1.36.0-wmf.3/extensions/CentralAuth/includes/CentralAuthUser.php(576): CentralAuthUser->loadState()
#11 /srv/mediawiki/php-1.36.0-wmf.3/extensions/CentralAuth/includes/CentralAuthUser.php(673): CentralAuthUser->getId()
#12 /srv/mediawiki/php-1.36.0-wmf.3/extensions/CentralAuth/includes/CentralAuthHooks.php(931): CentralAuthUser->exists()
#13 /srv/mediawiki/php-1.36.0-wmf.3/includes/HookContainer/HookContainer.php(320): CentralAuthHooks::onGetUserBlock(User, string, MediaWiki\Block\SystemBlock)
#14 /srv/mediawiki/php-1.36.0-wmf.3/includes/HookContainer/HookContainer.php(131): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#15 /srv/mediawiki/php-1.36.0-wmf.3/includes/HookContainer/HookRunner.php(1987): MediaWiki\HookContainer\HookContainer->run(string, array)
#16 /srv/mediawiki/php-1.36.0-wmf.3/includes/block/BlockManager.php(169): MediaWiki\HookContainer\HookRunner->onGetUserBlock(User, string, MediaWiki\Block\SystemBlock)
#17 /srv/mediawiki/php-1.36.0-wmf.3/includes/user/User.php(1604): MediaWiki\Block\BlockManager->getUserBlock(User, WebRequest, boolean)
#18 /srv/mediawiki/php-1.36.0-wmf.3/includes/user/User.php(1896): User->getBlockedStatus(boolean)
#19 /srv/mediawiki/php-1.36.0-wmf.3/includes/block/BlockManager.php(472): User->getBlock()
#20 /srv/mediawiki/php-1.36.0-wmf.3/includes/MediaWiki.php(767): MediaWiki\Block\BlockManager->trackBlockWithCookie(User, WebResponse)
#21 /srv/mediawiki/php-1.36.0-wmf.3/includes/api/ApiMain.php(555): MediaWiki::preOutputCommit(DerivativeContext)
#22 /srv/mediawiki/php-1.36.0-wmf.3/includes/api/ApiMain.php(500): ApiMain->executeActionWithErrorHandling()
#23 /srv/mediawiki/php-1.36.0-wmf.3/api.php(90): ApiMain->execute()
#24 /srv/mediawiki/php-1.36.0-wmf.3/api.php(45): wfApiMain()
#25 /srv/mediawiki/w/api.php(3): require(string)
#26 {main}

Re-tagging CPT. This is an active production error in an area we agreed CPT would take stewardship over. This isn't awaiting code review, this needs someone to investigate and fix the problem.

Still happening:

'metawiki.globaluser' doesn't exist (10.192.32.168)
Function: CentralAuthUser::loadFromDatabase
Query: SELECT  gu_id,gu_name,lu_wiki,gu_salt,gu_password,gu_auth_token,gu_locked,gu_hidden,gu_registration,gu_email,gu_email_authenticated,gu_home_db,gu_cas_token  FROM `globaluser` LEFT OUTER JOIN `localuser` ON ((gu_name=lu_name) AND lu_wiki = 'enwiki')   WHERE gu_name = 'xxxxxxx'  LIMIT 1
trace
#0 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1683): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1658): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1227): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1907): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(2007): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#5 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->selectRow(array, array, array, string, array, array)
#6 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(331): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.36.0-wmf.10/extensions/CentralAuth/includes/CentralAuthUser.php(472): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#8 /srv/mediawiki/php-1.36.0-wmf.10/extensions/CentralAuth/includes/CentralAuthUser.php(536): CentralAuthUser->loadFromDatabase()
#9 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1533): CentralAuthUser->{closure}(array, integer, array, double, array)
#10 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(2496): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#11 /srv/mediawiki/php-1.36.0-wmf.10/includes/deferred/MWCallableUpdate.php(38): WANObjectCache->{closure}()
#12 /srv/mediawiki/php-1.36.0-wmf.10/includes/deferred/DeferredUpdates.php(467): MWCallableUpdate->doUpdate()
#13 /srv/mediawiki/php-1.36.0-wmf.10/includes/deferred/DeferredUpdates.php(344): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#14 /srv/mediawiki/php-1.36.0-wmf.10/includes/deferred/DeferredUpdates.php(278): DeferredUpdates::run(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#15 /srv/mediawiki/php-1.36.0-wmf.10/includes/deferred/DeferredUpdates.php(194): DeferredUpdates::handleUpdateQueue(array, string, integer)
#16 /srv/mediawiki/php-1.36.0-wmf.10/includes/MediaWiki.php(1113): DeferredUpdates::doUpdates(string)
#17 /srv/mediawiki/php-1.36.0-wmf.10/includes/MediaWiki.php(849): MediaWiki->restInPeace()
#18 /srv/mediawiki/php-1.36.0-wmf.10/includes/MediaWiki.php(861): MediaWiki->{closure}()
#19 /srv/mediawiki/php-1.36.0-wmf.10/api.php(125): MediaWiki->doPostOutputShutdown()
#20 /srv/mediawiki/php-1.36.0-wmf.10/api.php(45): wfApiMain()
#21 /srv/mediawiki/w/api.php(3): require(string)
#22 {main}

I'm seeing timeout associated with these entries, e.g.:

the execution time limit of 60 seconds was exceeded

#0 /srv/mediawiki/php-1.36.0-wmf.16/includes/libs/rdbms/database/DatabaseMysqli.php(46): {closure}(integer)
#1 /srv/mediawiki/php-1.36.0-wmf.16/includes/libs/rdbms/database/Database.php(1376): Wikimedia\Rdbms\DatabaseMysqli->doQuery(string)
#2 /srv/mediawiki/php-1.36.0-wmf.16/includes/libs/rdbms/database/Database.php(1294): Wikimedia\Rdbms\Database->executeQueryAttempt(string, string, boolean, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.16/includes/libs/rdbms/database/DatabaseMysqlBase.php(204): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.16/includes/libs/rdbms/database/Database.php(2776): Wikimedia\Rdbms\DatabaseMysqlBase->doSelectDomain(Wikimedia\Rdbms\DatabaseDomain)
#5 /srv/mediawiki/php-1.36.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1187): Wikimedia\Rdbms\Database->selectDomain(Wikimedia\Rdbms\DatabaseDomain)
#6 /srv/mediawiki/php-1.36.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(928): Wikimedia\Rdbms\LoadBalancer->getForeignConnection(integer, string, integer)
#7 /srv/mediawiki/php-1.36.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(899): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#8 /srv/mediawiki/php-1.36.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1030): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#9 /srv/mediawiki/php-1.36.0-wmf.16/extensions/GlobalUserPage/includes/GlobalUserPage.php(225): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, string)
#10 /srv/mediawiki/php-1.36.0-wmf.16/extensions/GlobalUserPage/includes/GlobalUserPage.php(200): MediaWiki\GlobalUserPage\GlobalUserPage::getCentralTouched(User)
#11 /srv/mediawiki/php-1.36.0-wmf.16/extensions/GlobalUserPage/includes/Hooks.php(57): MediaWiki\GlobalUserPage\GlobalUserPage::shouldDisplayGlobalPage(Title)
#12 /srv/mediawiki/php-1.36.0-wmf.16/includes/HookContainer/HookContainer.php(333): MediaWiki\GlobalUserPage\Hooks::onTitleIsAlwaysKnown(Title, NULL)
#13 /srv/mediawiki/php-1.36.0-wmf.16/includes/HookContainer/HookContainer.php(140): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#14 /srv/mediawiki/php-1.36.0-wmf.16/includes/HookContainer/HookRunner.php(3946): MediaWiki\HookContainer\HookContainer->run(string, array)
#15 /srv/mediawiki/php-1.36.0-wmf.16/includes/Title.php(3962): MediaWiki\HookContainer\HookRunner->onTitleIsAlwaysKnown(Title, NULL)
#16 /srv/mediawiki/php-1.36.0-wmf.16/includes/parser/Parser.php(2670): Title->isAlwaysKnown()
#17 /srv/mediawiki/php-1.36.0-wmf.16/includes/parser/Parser.php(2366): Parser->handleInternalLinks2(string)
#18 /srv/mediawiki/php-1.36.0-wmf.16/includes/parser/Parser.php(1574): Parser->handleInternalLinks(string)
#19 /srv/mediawiki/php-1.36.0-wmf.16/includes/parser/Parser.php(646): Parser->internalParse(string)
#20 /srv/mediawiki/php-1.36.0-wmf.16/includes/content/WikitextContent.php(375): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#21 /srv/mediawiki/php-1.36.0-wmf.16/includes/content/AbstractContent.php(590): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#22 /srv/mediawiki/php-1.36.0-wmf.16/includes/Revision/RenderedRevision.php(266): AbstractContent->getParserOutput(Title, integer, ParserOptions, boolean)
#23 /srv/mediawiki/php-1.36.0-wmf.16/includes/Revision/RenderedRevision.php(235): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#24 /srv/mediawiki/php-1.36.0-wmf.16/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#25 /srv/mediawiki/php-1.36.0-wmf.16/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#26 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#27 /srv/mediawiki/php-1.36.0-wmf.16/includes/Revision/RenderedRevision.php(197): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#28 /srv/mediawiki/php-1.36.0-wmf.16/includes/poolcounter/PoolWorkArticleView.php(215): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#29 /srv/mediawiki/php-1.36.0-wmf.16/includes/poolcounter/PoolCounterWork.php(162): PoolWorkArticleView->doWork()
#30 /srv/mediawiki/php-1.36.0-wmf.16/includes/page/Article.php(811): PoolCounterWork->execute()
#31 /srv/mediawiki/php-1.36.0-wmf.16/includes/actions/ViewAction.php(74): Article->view()
#32 /srv/mediawiki/php-1.36.0-wmf.16/includes/MediaWiki.php(530): ViewAction->show()
#33 /srv/mediawiki/php-1.36.0-wmf.16/includes/MediaWiki.php(316): MediaWiki->performAction(Article, Title)
#34 /srv/mediawiki/php-1.36.0-wmf.16/includes/MediaWiki.php(943): MediaWiki->performRequest()
#35 /srv/mediawiki/php-1.36.0-wmf.16/includes/MediaWiki.php(546): MediaWiki->main()
#36 /srv/mediawiki/php-1.36.0-wmf.16/index.php(53): MediaWiki->run()
#37 /srv/mediawiki/php-1.36.0-wmf.16/index.php(46): wfIndexMain()
#38 /srv/mediawiki/w/index.php(3): require(string)
#39 {main}

Change 644636 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: detect corrupt Database instances due to critical section failure

https://gerrit.wikimedia.org/r/644636

Change 667751 had a related patch set uploaded (by Tim Starling; owner: Tim Starling):
[mediawiki/libs/RequestTimeout@master] Allow emergency timeouts to be overridden when creating the critical section

https://gerrit.wikimedia.org/r/667751

Change 667751 merged by jenkins-bot:
[mediawiki/libs/RequestTimeout@master] Allow emergency timeouts to be overridden when creating the critical section

https://gerrit.wikimedia.org/r/667751

Change 644636 merged by jenkins-bot:

[mediawiki/core@master] rdbms: detect corrupt Database instances due to critical section failure

https://gerrit.wikimedia.org/r/644636

Change 644636 merged by jenkins-bot:

[mediawiki/core@master] rdbms: detect corrupt Database instances due to critical section failure

https://gerrit.wikimedia.org/r/644636

Very rough analysis so far: Seemingly no difference. Is this expected?

This patch first went out in this weeks of 1.37.0-wmf.1. It does not appear to have solved or significantly changed the rate of Error: 1146 so far. With this week's wmf branch version having experience roughly the same number of these errors as the three wmf branches before it (quarter of the MW version pie chart).

https://logstash.wikimedia.org/goto/dccb5069d5f7b0efb3043a8dffe9d73e

The new edge case we're able to detect with this code seems to be hit:

[{reqId}] {exception_url} Wikimedia\Rdbms\DBUnexpectedError: Cannot execute Wikimedia\Rdbms\Database::runOnTransactionIdleCallbacks critical section while session state is out of sync. A critical section from Wikimedia\Rdbms\Database::executeQuery has

Couple of hits every few hours. Not much, but worth looking into perhaps:

https://logstash.wikimedia.org/goto/8f33a466461010cfd4cbd3d06df76063