Page MenuHomePhabricator

Special:GlobalRenameProgress fails with "Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'metawiki.renameuser_status' doesn't exist"
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'metawiki.renameuser_status' doesn't exist
Function: MediaWiki\Extension\CentralAuth\GlobalRename\GlobalRenameUserStatus::getInProgressRenames
Query: SELECT DISTINCT ru_oldname,ru
exception.trace
from /srv/mediawiki/php-1.40.0-wmf.17/includes/libs/rdbms/database/Database.php(1536)
#0 /srv/mediawiki/php-1.40.0-wmf.17/includes/libs/rdbms/database/Database.php(1520): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.40.0-wmf.17/includes/libs/rdbms/database/Database.php(1494): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.40.0-wmf.17/includes/libs/rdbms/database/Database.php(867): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.40.0-wmf.17/includes/libs/rdbms/database/Database.php(1628): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.40.0-wmf.17/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(652): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#5 /srv/mediawiki/php-1.40.0-wmf.17/extensions/CentralAuth/includes/GlobalRename/GlobalRenameUserStatus.php(252): Wikimedia\Rdbms\SelectQueryBuilder->fetchResultSet()
#6 /srv/mediawiki/php-1.40.0-wmf.17/extensions/CentralAuth/includes/Special/SpecialGlobalRenameProgress.php(71): MediaWiki\Extension\CentralAuth\GlobalRename\GlobalRenameUserStatus::getInProgressRenames(User)
#7 /srv/mediawiki/php-1.40.0-wmf.17/extensions/CentralAuth/includes/Special/SpecialGlobalRenameProgress.php(92): MediaWiki\Extension\CentralAuth\Special\SpecialGlobalRenameProgress->showCurrentRenames()
#8 /srv/mediawiki/php-1.40.0-wmf.17/includes/htmlform/HTMLForm.php(731): MediaWiki\Extension\CentralAuth\Special\SpecialGlobalRenameProgress->onSubmit(array, OOUIHTMLForm)
#9 /srv/mediawiki/php-1.40.0-wmf.17/includes/htmlform/HTMLForm.php(621): HTMLForm->trySubmit()
#10 /srv/mediawiki/php-1.40.0-wmf.17/includes/htmlform/HTMLForm.php(637): HTMLForm->tryAuthorizedSubmit()
#11 /srv/mediawiki/php-1.40.0-wmf.17/includes/specialpage/FormSpecialPage.php(211): HTMLForm->show()
#12 /srv/mediawiki/php-1.40.0-wmf.17/includes/specialpage/SpecialPage.php(700): FormSpecialPage->execute(NULL)
#13 /srv/mediawiki/php-1.40.0-wmf.17/includes/specialpage/SpecialPageFactory.php(1458): SpecialPage->run(NULL)
#14 /srv/mediawiki/php-1.40.0-wmf.17/includes/MediaWiki.php(322): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#15 /srv/mediawiki/php-1.40.0-wmf.17/includes/MediaWiki.php(916): MediaWiki->performRequest()
#16 /srv/mediawiki/php-1.40.0-wmf.17/includes/MediaWiki.php(571): MediaWiki->main()
#17 /srv/mediawiki/php-1.40.0-wmf.17/index.php(50): MediaWiki->run()
#18 /srv/mediawiki/php-1.40.0-wmf.17/index.php(46): wfIndexMain()
#19 /srv/mediawiki/w/index.php(3): require(string)
#20 {main}
Impact

https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress does not work.

Notes

GlobalRenameUserStatus talks to metawiki's database, when it should talk to centralauth's. Not clear why; it calls CentralAuthServices::getDatabaseManager()->getCentralDB( DB_REPLICA ), and when I run the exact same query in shell.php session, it...works.

Event Timeline

Localhost debugging:

urbanecm@notebook  ~/unsynced/gerrit/mediawiki/core
$ !php
php maintenance/shell.php --wiki=awiki

*******************************************************************************
NOTE: Do not run maintenance scripts directly, use maintenance/run.php instead!
      Running scripts directly has been deprecated in MediaWiki 1.40.
      It may not work for some (or any) scripts in the future.
*******************************************************************************

Psy Shell v0.11.8 (PHP 7.4.33 — cli) by Justin Hileman
>>> \MediaWiki\Extension\CentralAuth\GlobalRename\GlobalRenameUserStatus::getInProgressRenames(User::newFromName('Martin Urbanec'))
Wikimedia\Rdbms\DBQueryError with message 'Error 1146: Table 'awiki.renameuser_status' doesn't exist
Function: MediaWiki\Extension\CentralAuth\GlobalRename\GlobalRenameUserStatus::getInProgressRenames
Query: SELECT DISTINCT ru_oldname,ru_newname  FROM `renameuser_status`
'
>>> use MediaWiki\Extension\CentralAuth\CentralAuthServices;
>>> $dbr = CentralAuthServices::getDatabaseManager()->getCentralDB( DB_REPLICA );
=> Wikimedia\Rdbms\DBConnRef {#4034}

>>> $qb = $dbr->newSelectQueryBuilder();
=> Wikimedia\Rdbms\SelectQueryBuilder {#4639}

>>> $qb->select( [ 'ru_oldname', 'ru_newname' ] ) ->distinct()->from( 'renameuser_status' );
=> Wikimedia\Rdbms\SelectQueryBuilder {#4639}

>>> $res = $qb ->caller( __METHOD__ )->fetchResultSet();
=> Wikimedia\Rdbms\MysqliResultWrapper {#4619}

>>> $ret = [];
=> []

>>> foreach ( $res as $row ) { $ret[$row->ru_oldname] = $row->ru_newname;}
>>> $ret
=> []

>>>

You can see it talks to the wrong database when calling the method, but when copy pasting exactly the same code from the method definition, it works as expected.

Change 876050 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/CentralAuth@master] Revert "GlobalRename: Convert DB selects to use SelectQueryBuilder"

https://gerrit.wikimedia.org/r/876050

Change 876051 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/CentralAuth@wmf/1.40.0-wmf.17] Revert "GlobalRename: Convert DB selects to use SelectQueryBuilder"

https://gerrit.wikimedia.org/r/876051

Change 876051 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.40.0-wmf.17] Revert "GlobalRename: Convert DB selects to use SelectQueryBuilder"

https://gerrit.wikimedia.org/r/876051

Urbanecm renamed this task from Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'metawiki.renameuser_status' doesn't existFunction: MediaWiki\Extension\CentralAuth\GlobalRename\GlobalRenameUserStatus::getInProgressRenamesQuery: SELECT DISTINCT ru_oldname,ru to Special:GlobalRenameProgress fails with "Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'metawiki.renameuser_status' doesn't exist".Jan 6 2023, 12:58 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-06T00:59:14Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:876051|Revert "GlobalRename: Convert DB selects to use SelectQueryBuilder" (T326377 T312394)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-06T01:01:01Z] <urbanecm@deploy1002> urbanecm and urbanecm: Backport for [[gerrit:876051|Revert "GlobalRename: Convert DB selects to use SelectQueryBuilder" (T326377 T312394)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-06T01:08:03Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:876051|Revert "GlobalRename: Convert DB selects to use SelectQueryBuilder" (T326377 T312394)]] (duration: 08m 48s)

Urbanecm triaged this task as High priority.Jan 6 2023, 1:10 AM

This is now fixed in wmf.17 only (the patch does not pass CI on the master branch). Adding as a blocker to next week's train.

Tagging RDBMS (+the Performance team) as well, as it looks to be a bug somewhere within the library, rather than in the caller (CentralAuth). See T326377#8503696 for the symptoms of the issue.

I have triple checked the patch and I can't find a case where the db object is being sent incorrectly. Debugging.

Am I missing something in reproducing the error?

ladsgroup@mwmaint1002:~$ mwscript shell.php --wiki=metawiki
Psy Shell v0.11.8 (PHP 7.4.33 — cli) by Justin Hileman
>>> use MediaWiki\Extension\CentralAuth\CentralAuthServices;
>>> $dbr = CentralAuthServices::getDatabaseManager()->getCentralDB( DB_REPLICA );
=> Wikimedia\Rdbms\DBConnRef {#5868}

>>> $qb = $dbr->newSelectQueryBuilder();
=> Wikimedia\Rdbms\SelectQueryBuilder {#5879}

>>> $qb->select( [ 'ru_oldname', 'ru_newname' ] ) ->distinct()->from( 'renameuser_status' );
=> Wikimedia\Rdbms\SelectQueryBuilder {#5879}

>>> $res = $qb ->caller( __METHOD__ )->fetchResultSet();
=> Wikimedia\Rdbms\MysqliResultWrapper {#5849}

>>> $ret = [];
=> []

>>> foreach ( $res as $row ) { $ret[$row->ru_oldname] = $row->ru_newname;}
>>> $ret
=> [
     "Edegouges.olymp" => "Pot2000",
   ]

I can reproduce it in mwdebug. Debugging.

So far, the table name is set correctly.

I'm fairly certain at this point that this is T193565: Foreign query for metawiki fails with "Table 'centralauth.page' doesn't exist" (DBConnRef mixup?) which actually didn't get fixed. The reason is something along the lines of lb/lbf being initialized with metawiki db and lb reusing the metawiki connection because both are on s7 but failing to set the db correctly (probably setting it too late, e.g. after db prefix is added or something like that) and then exploding majestically. Trying to see what I can do to fix this.

Change 876222 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Stop tracking LBs based on section and track them based on domain

https://gerrit.wikimedia.org/r/876222

Merged the revert since we have no proper fix yet.

Change 876050 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Revert "GlobalRename: Convert DB selects to use SelectQueryBuilder"

https://gerrit.wikimedia.org/r/876050

I can reproduce it in mwdebug. Debugging.

So far, the table name is set correctly.

Did you find updated lines of code that I can put in shell.php/eval.php to reproduce?

I can reproduce it in mwdebug. Debugging.

So far, the table name is set correctly.

Did you find updated lines of code that I can put in shell.php/eval.php to reproduce?

It is in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/812552/7/includes/GlobalRename/GlobalRenameUserStatus.php (GlobalRenameUserStatus::getInProgressRenames).

But it won't work in shell.php/eval.php. You must change it in mwdebug and then go to https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress

The problem is from:

$qb = $dbr->newSelectQueryBuilder();
...
if ( !$performer->isAllowed( 'centralauth-suppress' ) ) {

The query build wraps the Database, not the DBConnRef, and isAllowed() is triggered a domain change on the Database handle. Once fetchResultSet() is called, there is nothing to switch the domain back.

I created an empty 'remotewiki' db to repro this pattern locally:

$dbr = wfGetDB( DB_REPLICA, [], false );
$qb = $dbr->newSelectQueryBuilder();
wfGetDB( DB_REPLICA, [], 'remotewiki' )->ensureConnection();
$qb->select( [ 'page_id' ] )->from( 'page' )->limit( 1 )->caller( __METHOD__ )->fetchResultSet();

Which gives:

Error 1146: Table 'remotewiki.page' doesn't exist

This works without the ensureConnection() line.

Change 881024 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] Use DBConnRef in SelectQueryBuilder

https://gerrit.wikimedia.org/r/881024

Change 881027 had a related patch set uploaded (by Krinkle; author: Tim Starling):

[mediawiki/core@REL1_38] rdbms: Use DBConnRef in SelectQueryBuilder

https://gerrit.wikimedia.org/r/881027

Change 881028 had a related patch set uploaded (by Krinkle; author: Tim Starling):

[mediawiki/core@REL1_39] rdbms: Use DBConnRef in SelectQueryBuilder

https://gerrit.wikimedia.org/r/881028

Change 881024 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Use DBConnRef in SelectQueryBuilder

https://gerrit.wikimedia.org/r/881024

Change 881027 merged by jenkins-bot:

[mediawiki/core@REL1_38] rdbms: Use DBConnRef in SelectQueryBuilder

https://gerrit.wikimedia.org/r/881027

Change 881028 merged by jenkins-bot:

[mediawiki/core@REL1_39] rdbms: Use DBConnRef in SelectQueryBuilder

https://gerrit.wikimedia.org/r/881028

Change 876222 abandoned by Ladsgroup:

[mediawiki/core@master] rdbms: Stop tracking LBs based on section and track them based on domain

Reason:

Done in a better way

https://gerrit.wikimedia.org/r/876222