Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'centralauth.logging' doesn't exist
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

service.version: 1.44.0-wmf.16
trace.id: 59f22818-86c7-4952-bbc8-765256c236e9
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'centralauth.logging' doesn't exist
Function: MediaWiki\CheckUser\HookHandler\CheckUserPrivateEventsHandler::onLocalUserCreated
Query: SELECT  log_id,log_type,log_action,log_times

Frame	Location	Call
from	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/database/Database.php(1230)
#0	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/database/Database.php(1214)	Wikimedia\Rdbms\Database->getQueryException(string, int, string, string)
#1	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/database/Database.php(1188)	Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, int, string, string)
#2	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/database/Database.php(647)	Wikimedia\Rdbms\Database->reportQueryError(string, int, string, string, bool)
#3	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/database/Database.php(1365)	Wikimedia\Rdbms\Database->query(Wikimedia\Rdbms\Query, string)
#4	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/database/Database.php(1374)	Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#5	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(810)	Wikimedia\Rdbms\Database->selectRow(array, array, array, string, array, array)
#6	/srv/mediawiki/php-1.44.0-wmf.16/extensions/CheckUser/src/HookHandler/CheckUserPrivateEventsHandler.php(127)	Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#7	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/database/TransactionManager.php(726)	MediaWiki\CheckUser\HookHandler\CheckUserPrivateEventsHandler->MediaWiki\CheckUser\HookHandler\{closure}(Wikimedia\Rdbms\DatabaseMySQL)
#8	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/database/Database.php(3303)	Wikimedia\Rdbms\TransactionManager->runOnTransactionPreCommitCallbacks(Wikimedia\Rdbms\DatabaseMySQL)
#9	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1357)	Wikimedia\Rdbms\Database->runOnTransactionPreCommitCallbacks()
#10	/srv/mediawiki/php-1.44.0-wmf.16/includes/libs/rdbms/lbfactory/LBFactory.php(322)	Wikimedia\Rdbms\LoadBalancer->finalizePrimaryChanges(string)
#11	/srv/mediawiki/php-1.44.0-wmf.16/extensions/EventBus/includes/JobExecutor.php(85)	Wikimedia\Rdbms\LBFactory->commitPrimaryChanges(string, int)
#12	/srv/mediawiki/rpc/RunSingleJob.php(60)	MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#13	{main}

Impact

300 errors in first 22 minutes of .16 deployed to group1.
Being unsure myself about user impact, I can still successfully log in on meta.

Notes

T384236#10541321 / https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1117927 might be related?

Details

Request URL: https://mw-jobrunner.discovery.wmnet/rpc/RunSingleJob.php

	Subject	Repo	Branch	Lines +/-
	Use original connection handle in onTransactionPreCommitOrIdle()	mediawiki/extensions/CheckUser	wmf/1.44.0-wmf.16	+1 -2
	Use original connection handle in onTransactionPreCommitOrIdle()	mediawiki/extensions/CheckUser	master	+1 -2

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	Aklapper	T382367 1.44.0-wmf.16 deployment blockers
		Resolved	PRODUCTION ERROR	mszabo	T386171 Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'centralauth.logging' doesn't exist

Event Timeline

Aklapper created this task.Feb 12 2025, 9:41 AM

Restricted Application added a project: Trust and Safety Product Team. · View Herald TranscriptFeb 12 2025, 9:41 AM

Aklapper updated the task description. (Show Details)Feb 12 2025, 9:44 AM

Aklapper mentioned this in T384236: [regression] Accounts created at test wikis are not receiving Growth features.Feb 12 2025, 9:53 AM

Aklapper updated the task description. (Show Details)

Dreamy_Jazz wrote on IRC "code triggering this is through the CentralAuthCreateLocalAccountJob from CentralAuth" thus re-tagging

Just to give some additional context:

CheckUser is getting the DB from IConnectionProvider::getPrimaryDatabase using the ConnectionProvider service, then using this for a IDatabase::onTransactionPreCommitOrIdle call, but somehow the DB then used inside this scope is not the DB for the local wiki
The auto-creation is caused by the CentralAuthCreateLocalAccountJob job from MediaWiki-extensions-CentralAuth, though that mostly just calls MediaWiki core auto-creation code
CheckUser recently was changed to wrap the failing code inside IDatabase::onTransactionPreCommitOrIdle, however, it is my understanding that the method should only run the code when the transaction is committed for the specific DB (and not any other DB).

AFACIS, this is either:

Somewhere some code is overriding the current database returned by the main ConnectionProvider service to be centralauth.
IDatabase::onTransactionPreCommitOrIdle is calling the code with a DB that is not the one it was registered with (i.e. call it with a local DB but have the code run using the central DB). However, if this is the case it seems like this should be a bug.

This usually happens on metawiki, but has also happened on frwiktionary too.

Aklapper mentioned this in T382367: 1.44.0-wmf.16 deployment blockers.Feb 12 2025, 10:25 AM

One idea I've got to fix this is that the IDatabase::onTransactionPreCommitOrIdle uses I see outside CheckUser do not use the IDatabase instance passed in the constructor. Perhaps this may be fixed by using the instance got from outside the ::onTransactionPreCommitOrIdle scope (like the other uses I find in codesearch).

However, I've got no way to really test if this would work apart from making this patch and applying it to production.

@mszabo, do you have any thoughts on this?

Dreamy_Jazz added a project: CheckUser.Feb 12 2025, 11:40 AM

Dreamy_Jazz added a project: Trust and Safety Product Team.

Dreamy_Jazz added a subscriber: mszabo.

Novem_Linguae subscribed.Feb 12 2025, 11:54 AM

Change #1119089 had a related patch set uploaded (by Máté Szabó; author: Máté Szabó):

[mediawiki/extensions/CheckUser@master] Use original connection handle in onTransactionPreCommitOrIdle()

https://gerrit.wikimedia.org/r/1119089

gerritbot added a project: Patch-For-Review.Feb 12 2025, 12:05 PM

Agusbou2015 subscribed.Feb 12 2025, 12:34 PM

Change #1119089 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Use original connection handle in onTransactionPreCommitOrIdle()

https://gerrit.wikimedia.org/r/1119089

Change #1119099 had a related patch set uploaded (by Máté Szabó; author: Máté Szabó):

[mediawiki/extensions/CheckUser@wmf/1.44.0-wmf.16] Use original connection handle in onTransactionPreCommitOrIdle()

https://gerrit.wikimedia.org/r/1119099

mszabo changed the task status from Open to In Progress.Feb 12 2025, 12:49 PM

mszabo claimed this task.

mszabo added a project: Trust and Safety Product Sprint (Sprint Hurdy-Gurdy (Feb 10 - Feb 28)).

T386190 filed to clean up the broken rdbms functionality.

Change #1119099 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@wmf/1.44.0-wmf.16] Use original connection handle in onTransactionPreCommitOrIdle()

https://gerrit.wikimedia.org/r/1119099

Mentioned in SAL (#wikimedia-operations) [2025-02-12T12:58:30Z] <mszabo@deploy2002> Started scap sync-world: Backport for [[gerrit:1119099|Use original connection handle in onTransactionPreCommitOrIdle() (T386171)]]

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.17; 2025-02-18).Feb 12 2025, 1:00 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-12T13:01:33Z] <mszabo@deploy2002> mszabo: Backport for [[gerrit:1119099|Use original connection handle in onTransactionPreCommitOrIdle() (T386171)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

mszabo moved this task from Priority Backlog to In Progress on the Trust and Safety Product Sprint (Sprint Hurdy-Gurdy (Feb 10 - Feb 28)) board.Feb 12 2025, 1:08 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-12T13:09:58Z] <mszabo@deploy2002> Finished scap sync-world: Backport for [[gerrit:1119099|Use original connection handle in onTransactionPreCommitOrIdle() (T386171)]] (duration: 11m 27s)