Page MenuHomePhabricator
Create Task
Maniphest T225682

Login, account creation, anything else that accesses global user throws DBReadOnlyRoleError in beta labs
Closed, ResolvedPublic
Actions

Description

For example, when I try to log in in beta labs:

[XQGFxqwQBHcAABxvtLIAAAAM] /wiki/Special:UserLogin Wikimedia\Rdbms\DBReadOnlyRoleError from line 1040 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: Write operations are not allowed on replica database connections.

Backtrace:

#0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(1249): Wikimedia\Rdbms\Database->assertIsWritableMaster()
#1 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(248): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#2 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(2412): Wikimedia\Rdbms\DatabaseMysqlBase->doSelectDomain(Wikimedia\Rdbms\DatabaseDomain)
#3 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1036): Wikimedia\Rdbms\Database->selectDomain(Wikimedia\Rdbms\DatabaseDomain)
#4 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(890): Wikimedia\Rdbms\LoadBalancer->openForeignConnection(integer, string, integer)
#5 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(763): Wikimedia\Rdbms\LoadBalancer->openConnection(integer, string, integer)
#6 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(850): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#7 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2485): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, string)
#8 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2381): CentralAuthUser->localUserData(string)
#9 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2354): CentralAuthUser->queryAttached()
#10 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthHooks.php(1433): CentralAuthUser->getLocalGroups()
#11 /srv/mediawiki/php-master/includes/Hooks.php(174): CentralAuthHooks::onPasswordPoliciesForUser(User, array)
#12 /srv/mediawiki/php-master/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#13 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(180): Hooks::run(string, array)
#14 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(77): UserPasswordPolicy->getPoliciesForUser(User)
#15 /srv/mediawiki/php-master/includes/user/User.php(1208): UserPasswordPolicy->checkUserPassword(User, string)
#16 /srv/mediawiki/php-master/includes/auth/AbstractPasswordPrimaryAuthenticationProvider.php(106): User->checkPasswordValidity(string)
#17 /srv/mediawiki/php-master/includes/auth/TemporaryPasswordPrimaryAuthenticationProvider.php(142): MediaWiki\Auth\AbstractPasswordPrimaryAuthenticationProvider->checkPasswordValidity(string, string)
#18 /srv/mediawiki/php-master/includes/auth/AuthManager.php(455): MediaWiki\Auth\TemporaryPasswordPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
#19 /srv/mediawiki/php-master/includes/auth/AuthManager.php(385): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#20 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(353): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#21 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(482): AuthManagerSpecialPage->performAuthenticationStep(string, array)
#22 /srv/mediawiki/php-master/includes/htmlform/HTMLForm.php(672): AuthManagerSpecialPage->handleFormSubmit(array, VFormHTMLForm)
#23 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(416): HTMLForm->trySubmit()
#24 /srv/mediawiki/php-master/includes/specialpage/LoginSignupSpecialPage.php(313): AuthManagerSpecialPage->trySubmit()
#25 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(571): LoginSignupSpecialPage->execute(NULL)
#26 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(575): SpecialPage->run(NULL)
#27 /srv/mediawiki/php-master/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#28 /srv/mediawiki/php-master/includes/MediaWiki.php(865): MediaWiki->performRequest()
#29 /srv/mediawiki/php-master/includes/MediaWiki.php(515): MediaWiki->main()
#30 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#31 /srv/mediawiki/w/index.php(3): include(string)
#32 {main}

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Database: Recognize USE queries as non-write queriesCatropemediawiki/coremaster+2 -2
Customize query in gerrit

Related Objects

Event Timeline

Catrope created this task.Jun 12 2019, 11:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 12 2019, 11:08 PM
Catrope triaged this task as Unbreak Now! priority.Jun 12 2019, 11:08 PM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptJun 12 2019, 11:08 PM
Catrope mentioned this in T225668: Homepage: start module layout on iOS Safari 10 and iOS Chrome 60.0.3112.89.Jun 12 2019, 11:09 PM

The first error of this type appeared at 20:05:59 UTC today (June 12)

Catrope added a comment.EditedJun 12 2019, 11:13 PM

I believe https://gerrit.wikimedia.org/r/c/mediawiki/core/+/512043 is likely to be the culprit.

This beta-scap-eqiad job began at 20:04:28 and synced to canaries from 20:05:52 to 20:05:58. The first error appeared at 20:05:59.

The job that triggered it was this beta-code-update-eqiad job, which reports an update range of 498df1b54e..7b5f218bd7, and the only commits in that range are this change and its merge commit. The change touches database files, so it seems plausible that it could be the cause of exceptions coming from the database layer.

Catrope added a subscriber: aaron.Jun 12 2019, 11:19 PM
Krenair subscribed.Jun 12 2019, 11:19 PM
Catrope added a comment.Jun 12 2019, 11:23 PM

I think this happened because the change made DatabaseMysqlBase::doSelectDomain() use executeQuery() whereas it previously used doQuery(). The former has the assertIsWritableMaster() check but the latter bypasses it. USE queries appear to be considered write queries (because they're not explicitly marked as read queries by the regex in Database::isWriteQuery()), so this failure started happening every time we try to switch databases in MySQL.

I will try to fix this by recognizing USE queries as non-write queries; if that doesn't work, I'll revert the change.

Catrope claimed this task.Jun 12 2019, 11:24 PM
Catrope added a comment.Jun 12 2019, 11:28 PM

Strangely, I can't reproduce this locally, even in a Vagrant setup with CentralAuth. Perhaps it's specific to having the master and replica be different servers.

gerritbot added a comment.Jun 12 2019, 11:33 PM

Change 516720 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/core@master] Database: Recognize USE queries as non-write queries

https://gerrit.wikimedia.org/r/516720

gerritbot added a project: Patch-For-Review.Jun 12 2019, 11:33 PM
gerritbot added a comment.Jun 12 2019, 11:54 PM

Change 516720 merged by jenkins-bot:
[mediawiki/core@master] Database: Recognize USE queries as non-write queries

https://gerrit.wikimedia.org/r/516720

Krinkle subscribed.Jun 12 2019, 11:59 PM

Also fatals at https://en.wikipedia.beta.wmflabs.org/wiki/Special:Contributions/10.0.0.1.

Should be fixed in 5-10min once it gets auto-deployed to Beta.

Maintenance_bot removed a project: Patch-For-Review.Jun 13 2019, 12:10 AM
Catrope closed this task as Resolved.Jun 13 2019, 12:43 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits