Page MenuHomePhabricator

DBPerformance warning: "Expectation masterConns <= 0 not met" from CentralAuth special pages
Open, Needs TriagePublic

Description

The DBPerformance contains several warnings about unexpected DB_MASTER connections on GET requests from CentralAuth special pages.

Logstash: https://logstash.wikimedia.org/goto/80e6ec068512bab1687d248131bbee51

  • CentralAuthGroupMembershipProxy
  • Special:GlobalRenameRequest
  • Special:CentralLogin
  • Special:GlobalRenameQueue
CentralAuthGroupMembershipProxy
  • Request ID: XW8QQwpAADkAAGvZccAAAAAF
  • Request URL: GET /w/index.php?user=Krinkle&title=Special%3AGlobalUserRights
Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1): [connect to 10.64.48.15 (centralauth)]
#5 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(269): CentralAuthUtils::getCentralDB()
#6 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(472): CentralAuthUser->getSafeReadDB()
#7 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(415): CentralAuthUser->loadFromDatabase()
#8 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(588): CentralAuthUser->loadState()
#9 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(685): CentralAuthUser->getId()
#10 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthGroupMembershipProxy.php(50): CentralAuthUser->exists()
#11 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalGroupMembership.php(112): CentralAuthGroupMembershipProxy::newFromName()
#12 /srv/mediawiki/php-1.34.0-wmf.20/includes/specials/SpecialUserrights.php(108): SpecialGlobalGroupMembership->fetchUser()
#13 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/SpecialPage.php(573): UserrightsPage->execute() …
#18 /srv/mediawiki/php-1.34.0-wmf.20/index.php(42): MediaWiki->run()

The caller doesn't appear to need a master DB (not sure?) and isn't specifically asking for it. But, it uses the CentralAuthGroupMembershipProxy class which is hardcoded to obtain a master connection. That seems problematic.

Special:GlobalRenameRequest
  • Request ID: XW8ZGQpAICAAACgl7RwAAABE
  • Request URL: GET /wiki/Special:GlobalRenameRequest
Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1): [connect to 10.64.48.15 (centralauth)]
#5 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(269): CentralAuthUtils::getCentralDB()
#6 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(472): CentralAuthUser->getSafeReadDB()
#7 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(415): CentralAuthUser->loadFromDatabase()
#8 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(588): CentralAuthUser->loadState()
#9 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(685): CentralAuthUser->getId()
#10 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/GlobalRename/GlobalRenameRequest.php(434): CentralAuthUser->exists()
#11 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameRequest.php(211): GlobalRenameRequest::isNameAvailable()
#12 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameRequest.php(152): SpecialGlobalRenameRequest->suggestedUsername()
#13 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/FormSpecialPage.php(121): SpecialGlobalRenameRequest->getFormFields()
#14 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/FormSpecialPage.php(183): FormSpecialPage->getForm()
#15 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameRequest.php(89): FormSpecialPage->execute()
#16 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/SpecialPage.php(573): SpecialGlobalRenameRequest->execute() …
#21 /srv/mediawiki/php-1.34.0-wmf.20/index.php(42): MediaWiki->run()

Upon opening the special page form (viewing, not submitting), it calls GlobalRenameRequest::isNameAvailable which is hardwired to obtain a master instance. If don't know if it relies on that, but either that method or the call to it needs to change.

Special:CentralLogin
  • Request ID: XW8VAgpAADkAAFJxY88AAABR
  • Request URL: loginwiki GET /wiki/Special:CentralLogin/start?token=…&cpPosIndex=1 (referer ja.wikipedia.org)
Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 2): [connect to 10.64.48.15 (centralauth)] …
#5 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(269): CentralAuthUtils::getCentralDB()
#6 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(472): CentralAuthUser->getSafeReadDB()
#7 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(415): CentralAuthUser->loadFromDatabase()
#8 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(588): CentralAuthUser->loadState()
#9 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(685): CentralAuthUser->getId()
#10 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialCentralLogin.php(79): CentralAuthUser->exists()
#11 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialCentralLogin.php(97): Closure$SpecialCentralLogin::doLoginStart()
#12 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialCentralLogin.php(50): SpecialCentralLogin->doLoginStart()
#13 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/SpecialPage.php(573): SpecialCentralLogin->execute() …
#18 /srv/mediawiki/php-1.34.0-wmf.20/index.php(42): MediaWiki->run()

In SpecialCentralLogin->doLoginStart, a check is made for the replica DB, and if the user is not found, it then retries from a master DB. This seems intentional, but I'm not sure it is still needed.

For one, chronology protector should be taking care of this already by ensuring we have a replica that has caught up with the events of the user sign up. But the, why is it reaching the else clause? Perhaps this is an error case where the user doesn't exist at all. If so, then it'd be preferable not to handle errors by burdening the master with an impossible query. Alternatively, if we do need this and it's working fine, it should probably opt-out from the TrxProfiler the same way that e.g. RollbackAction does.

Special:GlobalRenameQueue
  • Request ID: XW63AgpAAEUAAFT6tf8AAACL
  • Request URL: GET /wiki/Special:GlobalRenameQueue/request/ …<number>…
Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1): [connect to 10.64.48.15 (centralauth)] …
#5 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(269): CentralAuthUtils::getCentralDB()
#6 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(472): CentralAuthUser->getSafeReadDB()
#7 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(415): CentralAuthUser->loadFromDatabase()
#8 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(705): CentralAuthUser->loadState()
#9 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameUser.php(227): CentralAuthUser->isHidden()
#10 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameQueue.php(427): SpecialGlobalRenameUser->processAntiSpoofConflicts()
#11 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameQueue.php(244): SpecialGlobalRenameQueue->doShowProcessForm()
#12 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameQueue.php(74): SpecialGlobalRenameQueue->handleProcessRequest()
#13 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/SpecialPage.php(573): SpecialGlobalRenameQueue->execute() …
#19 /srv/mediawiki/w/index.php(3): include()

The SpecialGlobalRenameUser->processAntiSpoofConflicts method is hardwired to obtain a master connection. Either needs to change, support both, or not be used when displaying the page in question.

See also:

Event Timeline

Krinkle updated the task description. (Show Details)

Notes about some research I did into the CentralAuthGroupMembershipProxy/SpecialGlobalGroupMembership issue:

  • It's based on UserrightsPage/SpecialUserrights.php
  • ::fetchUser( string $username, bool $writing = true ) is used to create membership proxy objects, $writing is always true
    • The function is called even when you have not specified a username, it attempts to guard against ($username === '' but in those cases $username is null)
    • $writing isn't used on this implementation at all, and on SpecialUserrights the only use is to validate userrights-interwiki user right
    • UserrightsProxy/UserrightsPage also always creates DB master connections when doing anything with interwiki userrights
    • side note: why does ApiQueryUsers call this method? this should probably extracted into its own service?
  • I don't think it doesn't matter if we load the user rights from a replica when viewing the page, master connection is required when updating user rights

From the duplicate task:

There are a number of different triggers, but the stack traces all involve calls to CentralAuthUser::getSafeReadDB, which chooses primary or replica depending on whether changes have been made in the primary database, regardless of the HTTP request method:

/**
 * @return IDatabase Master or replica based on shouldUseMasterDB()
 * @throws CentralAuthReadOnlyError
 */
protected function getSafeReadDB() {
	return $this->shouldUseMasterDB()
		? CentralAuthUtils::getCentralDB()
		: CentralAuthUtils::getCentralReplicaDB();
}

/**
 * Get (and init if needed) the value of mFromMaster
 *
 * @return bool
 */
protected function shouldUseMasterDB() {
	if ( $this->mFromMaster === null ) {
		// Calls LoadBalancer::hasOrMadeRecentMasterChanges
		$this->mFromMaster = self::centralLBHasRecentMasterChanges();
	}

	return $this->mFromMaster;
}

Another example that triggers this warning involves the hook handler CentralAuthHooks::onGetUserBlock:

from /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/TransactionProfiler.php(444)
#0 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/TransactionProfiler.php(224): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, string, integer)
#1 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1010): Wikimedia\Rdbms\TransactionProfiler->recordConnection(string, string, boolean)
#2 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(964): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#3 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1103): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUtilityService.php(110): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, string)
#5 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUtils.php(36): CentralAuthUtilityService->getCentralDB()
#6 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(273): CentralAuthUtils::getCentralDB()
#7 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(2378): CentralAuthUser->getSafeReadDB()
#8 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(594): CentralAuthUser->loadAttached()
#9 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(573): CentralAuthUser->loadFromCacheObject(array)
#10 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(417): CentralAuthUser->loadFromCache()
#11 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(605): CentralAuthUser->loadState()
#12 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(702): CentralAuthUser->getId()
#13 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthHooks.php(958): CentralAuthUser->exists()
#14 /srv/mediawiki/php-1.37.0-wmf.6/includes/HookContainer/HookContainer.php(330): CentralAuthHooks::onGetUserBlock(User, string, NULL)
#15 /srv/mediawiki/php-1.37.0-wmf.6/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#16 /srv/mediawiki/php-1.37.0-wmf.6/includes/HookContainer/HookRunner.php(1961): MediaWiki\HookContainer\HookContainer->run(string, array)
#17 /srv/mediawiki/php-1.37.0-wmf.6/includes/block/BlockManager.php(175): MediaWiki\HookContainer\HookRunner->onGetUserBlock(User, string, NULL)
#18 /srv/mediawiki/php-1.37.0-wmf.6/includes/user/User.php(1611): MediaWiki\Block\BlockManager->getUserBlock(User, WebRequest, boolean, boolean)
#19 /srv/mediawiki/php-1.37.0-wmf.6/includes/user/User.php(1906): User->getBlockedStatus(boolean, boolean)
#20 /srv/mediawiki/php-1.37.0-wmf.6/includes/block/BlockManager.php(484): User->getBlock()
#21 /srv/mediawiki/php-1.37.0-wmf.6/includes/MediaWiki.php(780): MediaWiki\Block\BlockManager->trackBlockWithCookie(User, WebResponse)
#22 /srv/mediawiki/php-1.37.0-wmf.6/includes/MediaWiki.php(646): MediaWiki::preOutputCommit(RequestContext, Closure)
#23 /srv/mediawiki/php-1.37.0-wmf.6/includes/MediaWiki.php(940): MediaWiki->doPreOutputCommit(Closure)
#24 /srv/mediawiki/php-1.37.0-wmf.6/includes/MediaWiki.php(551): MediaWiki->main()
#25 /srv/mediawiki/php-1.37.0-wmf.6/index.php(53): MediaWiki->run()
#26 /srv/mediawiki/php-1.37.0-wmf.6/index.php(46): wfIndexMain()
#27 /srv/mediawiki/w/index.php(3): require(string)
#28 {main}

Change 754534 had a related patch set uploaded (by Majavah; author: Majavah):

[mediawiki/extensions/CentralAuth@master] AntiSpoof: use replica db for displaying conflicts

https://gerrit.wikimedia.org/r/754534

Change 754537 had a related patch set uploaded (by Majavah; author: Majavah):

[mediawiki/extensions/CentralAuth@master] GlobalRenameRequest: use replica db to validate suggestions

https://gerrit.wikimedia.org/r/754537

Change 754534 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] AntiSpoof: use replica db for displaying conflicts

https://gerrit.wikimedia.org/r/754534

Change 754537 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] GlobalRenameRequest: use replica db to validate suggestions

https://gerrit.wikimedia.org/r/754537