Login, account creation, anything else that accesses global user throws DBReadOnlyRoleError in beta labs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Catrope
	Jun 12 2019, 11:08 PM

Description

For example, when I try to log in in beta labs:

[XQGFxqwQBHcAABxvtLIAAAAM] /wiki/Special:UserLogin Wikimedia\Rdbms\DBReadOnlyRoleError from line 1040 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: Write operations are not allowed on replica database connections.

Backtrace:

#0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(1249): Wikimedia\Rdbms\Database->assertIsWritableMaster()
#1 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(248): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#2 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(2412): Wikimedia\Rdbms\DatabaseMysqlBase->doSelectDomain(Wikimedia\Rdbms\DatabaseDomain)
#3 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1036): Wikimedia\Rdbms\Database->selectDomain(Wikimedia\Rdbms\DatabaseDomain)
#4 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(890): Wikimedia\Rdbms\LoadBalancer->openForeignConnection(integer, string, integer)
#5 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(763): Wikimedia\Rdbms\LoadBalancer->openConnection(integer, string, integer)
#6 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(850): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#7 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2485): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, string)
#8 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2381): CentralAuthUser->localUserData(string)
#9 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2354): CentralAuthUser->queryAttached()
#10 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthHooks.php(1433): CentralAuthUser->getLocalGroups()
#11 /srv/mediawiki/php-master/includes/Hooks.php(174): CentralAuthHooks::onPasswordPoliciesForUser(User, array)
#12 /srv/mediawiki/php-master/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#13 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(180): Hooks::run(string, array)
#14 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(77): UserPasswordPolicy->getPoliciesForUser(User)
#15 /srv/mediawiki/php-master/includes/user/User.php(1208): UserPasswordPolicy->checkUserPassword(User, string)
#16 /srv/mediawiki/php-master/includes/auth/AbstractPasswordPrimaryAuthenticationProvider.php(106): User->checkPasswordValidity(string)
#17 /srv/mediawiki/php-master/includes/auth/TemporaryPasswordPrimaryAuthenticationProvider.php(142): MediaWiki\Auth\AbstractPasswordPrimaryAuthenticationProvider->checkPasswordValidity(string, string)
#18 /srv/mediawiki/php-master/includes/auth/AuthManager.php(455): MediaWiki\Auth\TemporaryPasswordPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
#19 /srv/mediawiki/php-master/includes/auth/AuthManager.php(385): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#20 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(353): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#21 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(482): AuthManagerSpecialPage->performAuthenticationStep(string, array)
#22 /srv/mediawiki/php-master/includes/htmlform/HTMLForm.php(672): AuthManagerSpecialPage->handleFormSubmit(array, VFormHTMLForm)
#23 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(416): HTMLForm->trySubmit()
#24 /srv/mediawiki/php-master/includes/specialpage/LoginSignupSpecialPage.php(313): AuthManagerSpecialPage->trySubmit()
#25 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(571): LoginSignupSpecialPage->execute(NULL)
#26 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(575): SpecialPage->run(NULL)
#27 /srv/mediawiki/php-master/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#28 /srv/mediawiki/php-master/includes/MediaWiki.php(865): MediaWiki->performRequest()
#29 /srv/mediawiki/php-master/includes/MediaWiki.php(515): MediaWiki->main()
#30 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#31 /srv/mediawiki/w/index.php(3): include(string)
#32 {main}

Details

	Subject	Repo	Branch	Lines +/-
	Database: Recognize USE queries as non-write queries	mediawiki/core	master	+2 -2

Customize query in gerrit

Related Objects

Mentioned In: T225668: Homepage: start module layout on iOS Safari 10 and iOS Chrome 60.0.3112.89
Mentioned Here: rMW498df1b54e9c: Merge "Remove unnecessary semi-colons"
rMW7b5f218bd716: Merge "rdbms: add Database::executeQuery() method for internal use"

Event Timeline

• Catrope created this task.Jun 12 2019, 11:08 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 12 2019, 11:08 PM

• Catrope triaged this task as Unbreak Now! priority.Jun 12 2019, 11:08 PM

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptJun 12 2019, 11:08 PM

The first error of this type appeared at 20:05:59 UTC today (June 12)

I believe https://gerrit.wikimedia.org/r/c/mediawiki/core/+/512043 is likely to be the culprit.

This beta-scap-eqiad job began at 20:04:28 and synced to canaries from 20:05:52 to 20:05:58. The first error appeared at 20:05:59.

The job that triggered it was this beta-code-update-eqiad job, which reports an update range of 498df1b54e..7b5f218bd7, and the only commits in that range are this change and its merge commit. The change touches database files, so it seems plausible that it could be the cause of exceptions coming from the database layer.

• Catrope added a subscriber: aaron.Jun 12 2019, 11:19 PM

Krenair subscribed.Jun 12 2019, 11:19 PM

I think this happened because the change made DatabaseMysqlBase::doSelectDomain() use executeQuery() whereas it previously used doQuery(). The former has the assertIsWritableMaster() check but the latter bypasses it. USE queries appear to be considered write queries (because they're not explicitly marked as read queries by the regex in Database::isWriteQuery()), so this failure started happening every time we try to switch databases in MySQL.

I will try to fix this by recognizing USE queries as non-write queries; if that doesn't work, I'll revert the change.

• Catrope claimed this task.Jun 12 2019, 11:24 PM

Strangely, I can't reproduce this locally, even in a Vagrant setup with CentralAuth. Perhaps it's specific to having the master and replica be different servers.

Change 516720 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/core@master] Database: Recognize USE queries as non-write queries

https://gerrit.wikimedia.org/r/516720

gerritbot added a project: Patch-For-Review.Jun 12 2019, 11:33 PM

Change 516720 merged by jenkins-bot:
[mediawiki/core@master] Database: Recognize USE queries as non-write queries

https://gerrit.wikimedia.org/r/516720