Page MenuHomePhabricator

Login, account creation, anything else that accesses global user throws DBReadOnlyRoleError in beta labs
Closed, ResolvedPublic

Description

For example, when I try to log in in beta labs:

[XQGFxqwQBHcAABxvtLIAAAAM] /wiki/Special:UserLogin Wikimedia\Rdbms\DBReadOnlyRoleError from line 1040 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: Write operations are not allowed on replica database connections.

Backtrace:

#0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(1249): Wikimedia\Rdbms\Database->assertIsWritableMaster()
#1 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(248): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#2 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(2412): Wikimedia\Rdbms\DatabaseMysqlBase->doSelectDomain(Wikimedia\Rdbms\DatabaseDomain)
#3 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1036): Wikimedia\Rdbms\Database->selectDomain(Wikimedia\Rdbms\DatabaseDomain)
#4 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(890): Wikimedia\Rdbms\LoadBalancer->openForeignConnection(integer, string, integer)
#5 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(763): Wikimedia\Rdbms\LoadBalancer->openConnection(integer, string, integer)
#6 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(850): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#7 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2485): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, string)
#8 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2381): CentralAuthUser->localUserData(string)
#9 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2354): CentralAuthUser->queryAttached()
#10 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthHooks.php(1433): CentralAuthUser->getLocalGroups()
#11 /srv/mediawiki/php-master/includes/Hooks.php(174): CentralAuthHooks::onPasswordPoliciesForUser(User, array)
#12 /srv/mediawiki/php-master/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#13 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(180): Hooks::run(string, array)
#14 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(77): UserPasswordPolicy->getPoliciesForUser(User)
#15 /srv/mediawiki/php-master/includes/user/User.php(1208): UserPasswordPolicy->checkUserPassword(User, string)
#16 /srv/mediawiki/php-master/includes/auth/AbstractPasswordPrimaryAuthenticationProvider.php(106): User->checkPasswordValidity(string)
#17 /srv/mediawiki/php-master/includes/auth/TemporaryPasswordPrimaryAuthenticationProvider.php(142): MediaWiki\Auth\AbstractPasswordPrimaryAuthenticationProvider->checkPasswordValidity(string, string)
#18 /srv/mediawiki/php-master/includes/auth/AuthManager.php(455): MediaWiki\Auth\TemporaryPasswordPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
#19 /srv/mediawiki/php-master/includes/auth/AuthManager.php(385): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#20 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(353): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#21 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(482): AuthManagerSpecialPage->performAuthenticationStep(string, array)
#22 /srv/mediawiki/php-master/includes/htmlform/HTMLForm.php(672): AuthManagerSpecialPage->handleFormSubmit(array, VFormHTMLForm)
#23 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(416): HTMLForm->trySubmit()
#24 /srv/mediawiki/php-master/includes/specialpage/LoginSignupSpecialPage.php(313): AuthManagerSpecialPage->trySubmit()
#25 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(571): LoginSignupSpecialPage->execute(NULL)
#26 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(575): SpecialPage->run(NULL)
#27 /srv/mediawiki/php-master/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#28 /srv/mediawiki/php-master/includes/MediaWiki.php(865): MediaWiki->performRequest()
#29 /srv/mediawiki/php-master/includes/MediaWiki.php(515): MediaWiki->main()
#30 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#31 /srv/mediawiki/w/index.php(3): include(string)
#32 {main}

Event Timeline

Catrope triaged this task as Unbreak Now! priority.Jun 12 2019, 11:08 PM

I believe https://gerrit.wikimedia.org/r/c/mediawiki/core/+/512043 is likely to be the culprit.

This beta-scap-eqiad job began at 20:04:28 and synced to canaries from 20:05:52 to 20:05:58. The first error appeared at 20:05:59.

The job that triggered it was this beta-code-update-eqiad job, which reports an update range of 498df1b54e..7b5f218bd7, and the only commits in that range are this change and its merge commit. The change touches database files, so it seems plausible that it could be the cause of exceptions coming from the database layer.

I think this happened because the change made DatabaseMysqlBase::doSelectDomain() use executeQuery() whereas it previously used doQuery(). The former has the assertIsWritableMaster() check but the latter bypasses it. USE queries appear to be considered write queries (because they're not explicitly marked as read queries by the regex in Database::isWriteQuery()), so this failure started happening every time we try to switch databases in MySQL.

I will try to fix this by recognizing USE queries as non-write queries; if that doesn't work, I'll revert the change.

Strangely, I can't reproduce this locally, even in a Vagrant setup with CentralAuth. Perhaps it's specific to having the master and replica be different servers.

Change 516720 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/core@master] Database: Recognize USE queries as non-write queries

https://gerrit.wikimedia.org/r/516720

Change 516720 merged by jenkins-bot:
[mediawiki/core@master] Database: Recognize USE queries as non-write queries

https://gerrit.wikimedia.org/r/516720

Also fatals at https://en.wikipedia.beta.wmflabs.org/wiki/Special:Contributions/10.0.0.1.

Should be fixed in 5-10min once it gets auto-deployed to Beta.