Page MenuHomePhabricator
Create Task
Maniphest T263530

Table 'metawiki.globaluser' doesn't exist on query
Closed, DuplicatePublic
Actions

Description

For the last few days I have been seen errors from time to time like this (44 errors in the last 24h):

error	   	Table 'metawiki.globaluser' doesn't exist (10.192.0.146)
t exception.class	   	RuntimeException
t exception.code	   	0
t exception.file	   	/srv/mediawiki/php-1.36.0-wmf.9/includes/libs/rdbms/database/Database.php:1680
t exception.message	   	
t exception.trace	   	#0 /srv/mediawiki/php-1.36.0-wmf.9/includes/libs/rdbms/database/Database.php(1658): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.9/includes/libs/rdbms/database/Database.php(1227): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /srv/mediawiki/php-1.36.0-wmf.9/includes/libs/rdbms/database/Database.php(1907): Wikimedia\Rdbms\Database->query(string, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.9/includes/libs/rdbms/database/Database.php(2007): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#4 /srv/mediawiki/php-1.36.0-wmf.9/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->selectRow(array, array, array, string, array, array)
#5 /srv/mediawiki/php-1.36.0-wmf.9/includes/libs/rdbms/database/DBConnRef.php(331): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.36.0-wmf.9/extensions/CentralAuth/includes/CentralAuthUser.php(470): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#7 /srv/mediawiki/php-1.36.0-wmf.9/extensions/CentralAuth/includes/CentralAuthUser.php(534): CentralAuthUser->loadFromDatabase()
#8 /srv/mediawiki/php-1.36.0-wmf.9/includes/libs/objectcache/wancache/WANObjectCache.php(1533): CentralAuthUser->{closure}(array, integer, array, double, array)
#9 /srv/mediawiki/php-1.36.0-wmf.9/includes/libs/objectcache/wancache/WANObjectCache.php(2496): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#10 /srv/mediawiki/php-1.36.0-wmf.9/includes/deferred/MWCallableUpdate.php(38): WANObjectCache->{closure}()
#11 /srv/mediawiki/php-1.36.0-wmf.9/includes/deferred/DeferredUpdates.php(467): MWCallableUpdate->doUpdate()
#12 /srv/mediawiki/php-1.36.0-wmf.9/includes/deferred/DeferredUpdates.php(344): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#13 /srv/mediawiki/php-1.36.0-wmf.9/includes/deferred/DeferredUpdates.php(278): DeferredUpdates::run(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#14 /srv/mediawiki/php-1.36.0-wmf.9/includes/deferred/DeferredUpdates.php(194): DeferredUpdates::handleUpdateQueue(array, string, integer)
#15 /srv/mediawiki/php-1.36.0-wmf.9/includes/MediaWiki.php(1113): DeferredUpdates::doUpdates(string)
#16 /srv/mediawiki/php-1.36.0-wmf.9/includes/MediaWiki.php(849): MediaWiki->restInPeace()
#17 /srv/mediawiki/php-1.36.0-wmf.9/includes/MediaWiki.php(861): MediaWiki->{closure}()
#18 /srv/mediawiki/php-1.36.0-wmf.9/includes/MediaWiki.php(582): MediaWiki->doPostOutputShutdown()
#19 /srv/mediawiki/php-1.36.0-wmf.9/index.php(53): MediaWiki->run()
#20 /srv/mediawiki/php-1.36.0-wmf.9/index.php(46): wfIndexMain()
#21 /srv/mediawiki/w/index.php(3): require(string)
#22 {main}
t facility	   	user
t fname	   	CentralAuthUser::loadFromDatabase
t host	   	mw2272
t http_method	   	GET
t ip	   	10.192.48.94
t level	   	ERROR
t logsource	   	mw2272
t message	   	Error 1146 from CentralAuthUser::loadFromDatabase, Table 'metawiki.globaluser' doesn't exist (10.192.0.146) SELECT  gu_id,gu_name,lu_wiki,gu_salt,gu_password,gu_auth_token,gu_locked,gu_hidden,gu_registration,gu_email,gu_email_authenticated,gu_home_db,gu_cas_token  FROM `globaluser` LEFT OUTER JOIN `localuser` ON ((gu_name=lu_name) AND lu_wiki = 'enwiki')   WHERE gu_name = 'Goldentzofficial'  LIMIT 1   10.192.0.146

For some reason something is expecting to have globaluser table on metawiki, and that table only lives in centralauth.

Related Objects

Event Timeline

Marostegui created this task.Sep 22 2020, 10:28 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 22 2020, 10:28 AM
Marostegui added a comment.Sep 22 2020, 10:29 AM

Not really sure which tags to use here

LSobanski added a subscriber: LSobanski.Sep 22 2020, 10:55 AM
MarcoAurelio added a subscriber: MarcoAurelio.EditedSep 22 2020, 2:47 PM

Looks like the query at Line 286 at https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/CentralAuth/+/refs/heads/master/includes/CentralAuthUser.php

DannyS712 added a subscriber: DannyS712.Sep 22 2020, 3:53 PM
Reedy added a comment.Sep 22 2020, 4:22 PM

Do we know which wiki it's coming from? Or is it metawiki itself?

MarcoAurelio renamed this task from Table 'metawiki.globaluser' doesn't exist on query. to Table 'metawiki.globaluser' doesn't exist on query.Sep 22 2020, 7:54 PM
Reedy added a project: Wikimedia-production-error.Sep 22 2020, 7:59 PM
Reedy added a comment.Sep 22 2020, 8:04 PM
In T263530#6484325, @Reedy wrote:

Do we know which wiki it's coming from? Or is it metawiki itself?

Not that we vary $wgCentralAuthDatabase anyway in code or config...

So I'm presuming this code is hiccuping?

		$lbFactory = MediaWikiServices::getInstance()->getDBLoadBalancerFactory();
		return $lbFactory->getMainLB( $wgCentralAuthDatabase )
			->getConnectionRef( DB_REPLICA, 'centralauth', $wgCentralAuthDatabase );

Noting 'centralauth' is a group... Has anything changed with that recently? I did see T263127: Remove groups from db configs. but seemingly nothing actually actioned on that yet?

daniel added a subscriber: daniel.Sep 22 2020, 8:10 PM

Probably the same as T193565: Foreign query for metawiki fails with "Table 'centralauth.page' doesn't exist" (DBConnRef mixup?)

Krinkle added a comment.EditedSep 22 2020, 8:11 PM

It is indeed exactly the same. CA code and db config are all fine. The issue is rdbms mixing up connection objects.

Krinkle closed this task as a duplicate of T193565: Foreign query for metawiki fails with "Table 'centralauth.page' doesn't exist" (DBConnRef mixup?).Sep 22 2020, 8:11 PM
daniel triaged this task as High priority.Sep 22 2020, 8:11 PM
daniel added a project: Platform Team Workboards (Clinic Duty Team).
daniel moved this task from Inbox to Later on the Platform Team Workboards (Clinic Duty Team) board.
daniel removed a project: Platform Engineering.
Marostegui added a comment.Sep 23 2020, 5:10 AM
In T263530#6484325, @Reedy wrote:

Do we know which wiki it's coming from? Or is it metawiki itself?

enwiki from what I am seeing on logstash

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL