Page MenuHomePhabricator

Wikimedia\NormalizedException\NormalizedException: User ID: {centralUserId} mismatch with {storedUserId} for user: {username}
Open, Needs TriagePublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   RuntimeException: User ID mismatch
FrameLocationCall
from/srv/mediawiki/php-1.44.0-wmf.19/extensions/CentralAuth/includes/CentralAuthRedirectingPrimaryAuthenticationProvider.php(193)
#0/srv/mediawiki/php-1.44.0-wmf.19/includes/auth/AuthManager.php(638)MediaWiki\Extension\CentralAuth\CentralAuthRedirectingPrimaryAuthenticationProvider->continuePrimaryAuthentication(array)
#1/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(401)MediaWiki\Auth\AuthManager->continueAuthentication(array)
#2/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(533)MediaWiki\SpecialPage\AuthManagerSpecialPage->performAuthenticationStep(string, array)
#3/srv/mediawiki/php-1.44.0-wmf.19/includes/htmlform/HTMLForm.php(824)MediaWiki\SpecialPage\AuthManagerSpecialPage->handleFormSubmit(array, MediaWiki\HTMLForm\CodexHTMLForm)
#4/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(464)MediaWiki\HTMLForm\HTMLForm->trySubmit()
#5/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/LoginSignupSpecialPage.php(404)MediaWiki\SpecialPage\AuthManagerSpecialPage->trySubmit()
#6/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/SpecialPage.php(729)MediaWiki\SpecialPage\LoginSignupSpecialPage->execute(null)
#7/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/SpecialPageFactory.php(1737)MediaWiki\SpecialPage\SpecialPage->run(null)
#8/srv/mediawiki/php-1.44.0-wmf.19/includes/actions/ActionEntryPoint.php(503)MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
#9/srv/mediawiki/php-1.44.0-wmf.19/includes/actions/ActionEntryPoint.php(145)MediaWiki\Actions\ActionEntryPoint->performRequest()
#10/srv/mediawiki/php-1.44.0-wmf.19/includes/MediaWikiEntryPoint.php(202)MediaWiki\Actions\ActionEntryPoint->execute()
#11/srv/mediawiki/php-1.44.0-wmf.19/index.php(58)MediaWiki\MediaWikiEntryPoint->run()
#12/srv/mediawiki/w/index.php(3)require(string)
#13{main}
Impact

~10 people per day. Probably breaks autologin attempt and next autologin will succeed, so minimal.

Notes

This is some variant of T380500: CentralAuthUser returning outdated data after user creation.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Tgr subscribed.

Only a single instance (logstash). Do we seriously not log the user name/ID when this happens? Ugh.
It's very much not supposed to happen, but without any specifics, can't really investigate.

Change #1125389 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@master] Add user IDs that mismatch in the runtime exception

https://gerrit.wikimedia.org/r/1125389

Only a single instance (logstash). Do we seriously not log the user name/ID when this happens? Ugh.
It's very much not supposed to happen, but without any specifics, can't really investigate.

Add some logging and more info to the runtime exception is thrown.

Maybe one of the ids is zero and this is some version of T380500: CentralAuthUser returning outdated data after user creation.

Yes, I highly suspect we're hitting that. But let's find try to prove ourselves with some logging :)

Change #1125389 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Log and add user IDs that mismatch in the runtime exception

https://gerrit.wikimedia.org/r/1125389

Only a single instance (logstash).

Error
normalized_message
[{reqId}] {exception_url}   RuntimeException: User ID mismatch
FrameLocationCall
from/srv/mediawiki/php-1.44.0-wmf.19/extensions/CentralAuth/includes/CentralAuthRedirectingPrimaryAuthenticationProvider.php(193)
#0/srv/mediawiki/php-1.44.0-wmf.19/includes/auth/AuthManager.php(638)MediaWiki\Extension\CentralAuth\CentralAuthRedirectingPrimaryAuthenticationProvider->continuePrimaryAuthentication(array)
#1/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(401)MediaWiki\Auth\AuthManager->continueAuthentication(array)
#2/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(533)MediaWiki\SpecialPage\AuthManagerSpecialPage->performAuthenticationStep(string, array)
#3/srv/mediawiki/php-1.44.0-wmf.19/includes/htmlform/HTMLForm.php(824)MediaWiki\SpecialPage\AuthManagerSpecialPage->handleFormSubmit(array, MediaWiki\HTMLForm\CodexHTMLForm)
#4/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(464)MediaWiki\HTMLForm\HTMLForm->trySubmit()
#5/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/LoginSignupSpecialPage.php(404)MediaWiki\SpecialPage\AuthManagerSpecialPage->trySubmit()
#6/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/SpecialPage.php(729)MediaWiki\SpecialPage\LoginSignupSpecialPage->execute(null)
#7/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/SpecialPageFactory.php(1737)MediaWiki\SpecialPage\SpecialPage->run(null)
#8/srv/mediawiki/php-1.44.0-wmf.19/includes/actions/ActionEntryPoint.php(503)MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
#9/srv/mediawiki/php-1.44.0-wmf.19/includes/actions/ActionEntryPoint.php(145)MediaWiki\Actions\ActionEntryPoint->performRequest()
#10/srv/mediawiki/php-1.44.0-wmf.19/includes/MediaWikiEntryPoint.php(202)MediaWiki\Actions\ActionEntryPoint->execute()
#11/srv/mediawiki/php-1.44.0-wmf.19/index.php(58)MediaWiki\MediaWikiEntryPoint->run()
#12/srv/mediawiki/w/index.php(3)require(string)
#13{main}
Impact
Notes

Thanks @Reedy, we've added some logging, let's see if we can get more info about this as the patch goes out.

Change #1126051 had a related patch set uploaded (by Gergő Tisza; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.19] Log and add user IDs that mismatch in the runtime exception

https://gerrit.wikimedia.org/r/1126051

Change #1126051 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.19] Log and add user IDs that mismatch in the runtime exception

https://gerrit.wikimedia.org/r/1126051

Mentioned in SAL (#wikimedia-operations) [2025-03-10T14:49:53Z] <tgr@deploy2002> tgr: Backport for [[gerrit:1126050|SpecialCentralAutoLogin: Handle nullable wiki ID (T388252)]], [[gerrit:1126049|SUL3: Attach SUL mode to the return URL of local wiki (T388067)]], [[gerrit:1126051|Log and add user IDs that mismatch in the runtime exception (T388177)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-03-10T14:58:12Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1126050|SpecialCentralAutoLogin: Handle nullable wiki ID (T388252)]], [[gerrit:1126049|SUL3: Attach SUL mode to the return URL of local wiki (T388067)]], [[gerrit:1126051|Log and add user IDs that mismatch in the runtime exception (T388177)]] (duration: 15m 48s)

Both in practice seem to be against user id 0

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\NormalizedException\NormalizedException: User ID: {centralUserId} mismatch with {storedUserId} for user: {username}
FrameLocationCall
from/srv/mediawiki/php-1.44.0-wmf.19/extensions/CentralAuth/includes/CentralAuthRedirectingPrimaryAuthenticationProvider.php(199)
#0/srv/mediawiki/php-1.44.0-wmf.19/includes/auth/AuthManager.php(638)MediaWiki\Extension\CentralAuth\CentralAuthRedirectingPrimaryAuthenticationProvider->continuePrimaryAuthentication(array)
#1/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(401)MediaWiki\Auth\AuthManager->continueAuthentication(array)
#2/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(533)MediaWiki\SpecialPage\AuthManagerSpecialPage->performAuthenticationStep(string, array)
#3/srv/mediawiki/php-1.44.0-wmf.19/includes/htmlform/HTMLForm.php(824)MediaWiki\SpecialPage\AuthManagerSpecialPage->handleFormSubmit(array, MediaWiki\HTMLForm\CodexHTMLForm)
#4/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(464)MediaWiki\HTMLForm\HTMLForm->trySubmit()
#5/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/LoginSignupSpecialPage.php(404)MediaWiki\SpecialPage\AuthManagerSpecialPage->trySubmit()
#6/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/SpecialPage.php(729)MediaWiki\SpecialPage\LoginSignupSpecialPage->execute(null)
#7/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/SpecialPageFactory.php(1737)MediaWiki\SpecialPage\SpecialPage->run(null)
#8/srv/mediawiki/php-1.44.0-wmf.19/includes/actions/ActionEntryPoint.php(503)MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
#9/srv/mediawiki/php-1.44.0-wmf.19/includes/actions/ActionEntryPoint.php(145)MediaWiki\Actions\ActionEntryPoint->performRequest()
#10/srv/mediawiki/php-1.44.0-wmf.19/includes/MediaWikiEntryPoint.php(202)MediaWiki\Actions\ActionEntryPoint->execute()
#11/srv/mediawiki/php-1.44.0-wmf.19/index.php(58)MediaWiki\MediaWikiEntryPoint->run()
#12/srv/mediawiki/w/index.php(3)require(string)
#13{main}
Impact
Notes
Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\NormalizedException\NormalizedException: User ID: {centralUserId} mismatch with {storedUserId} for user: {username}
FrameLocationCall
from/srv/mediawiki/php-1.44.0-wmf.19/extensions/CentralAuth/includes/CentralAuthRedirectingPrimaryAuthenticationProvider.php(199)
#0/srv/mediawiki/php-1.44.0-wmf.19/includes/auth/AuthManager.php(638)MediaWiki\Extension\CentralAuth\CentralAuthRedirectingPrimaryAuthenticationProvider->continuePrimaryAuthentication(array)
#1/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(401)MediaWiki\Auth\AuthManager->continueAuthentication(array)
#2/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(533)MediaWiki\SpecialPage\AuthManagerSpecialPage->performAuthenticationStep(string, array)
#3/srv/mediawiki/php-1.44.0-wmf.19/includes/htmlform/HTMLForm.php(824)MediaWiki\SpecialPage\AuthManagerSpecialPage->handleFormSubmit(array, MediaWiki\HTMLForm\CodexHTMLForm)
#4/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(464)MediaWiki\HTMLForm\HTMLForm->trySubmit()
#5/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/LoginSignupSpecialPage.php(404)MediaWiki\SpecialPage\AuthManagerSpecialPage->trySubmit()
#6/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/SpecialPage.php(729)MediaWiki\SpecialPage\LoginSignupSpecialPage->execute(null)
#7/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/SpecialPageFactory.php(1737)MediaWiki\SpecialPage\SpecialPage->run(null)
#8/srv/mediawiki/php-1.44.0-wmf.19/includes/actions/ActionEntryPoint.php(503)MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
#9/srv/mediawiki/php-1.44.0-wmf.19/includes/actions/ActionEntryPoint.php(145)MediaWiki\Actions\ActionEntryPoint->performRequest()
#10/srv/mediawiki/php-1.44.0-wmf.19/includes/MediaWikiEntryPoint.php(202)MediaWiki\Actions\ActionEntryPoint->execute()
#11/srv/mediawiki/php-1.44.0-wmf.19/index.php(58)MediaWiki\MediaWikiEntryPoint->run()
#12/srv/mediawiki/w/index.php(3)require(string)
#13{main}
Impact
Notes

Sorry, had too many open windows.

30 examples in the last 7 days. The central used ID is always 0 (i.e. this is indeed caused by T380500: CentralAuthUser returning outdated data after user creation).

This always seems to be happening on the local continueAuthentication() call, presumably right after account creation.

Tgr renamed this task from RuntimeException: User ID mismatch to Wikimedia\NormalizedException\NormalizedException: User ID: {centralUserId} mismatch with {storedUserId} for user: {username}.Thu, Mar 20, 1:20 PM