Error during MFA setup for wikitech.wikimedia.org: MWException: CAS update failed on user_touched. The version of the user to be saved is older than the current version.
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	• WMDE_Norman
	Apr 4 2023, 9:49 AM

Description

Steps to replicate the issue (include links if applicable):

Login into Wikitech (using username + password)
in the settings click on Multi-factor-auth
provide username + password again to configure the MFA
get the following error message: [0030bbeb-12ea-4880-bbce-1e4f25226a58] 2023-04-04 07:56:24: Fatal exception of type "MWException" (instead of the configuration dialog)

What happens?:

get the following error message: [0030bbeb-12ea-4880-bbce-1e4f25226a58] 2023-04-04 07:56:24: Fatal exception of type "MWException" when trying to login with username + password

What should have happened instead?:

the configuration dialog should give me possibilities to configure the MFA (or to cancel it)

Software version (skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

I am using Firefox 111.0

Error

mwversion: 1.41.0-wmf.2
reqId: 0030bbeb-12ea-4880-bbce-1e4f25226a58
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   MWException: CAS update failed on user_touched. The version of the user to be saved is older than the current version.

exception.trace

from /srv/mediawiki/php-1.41.0-wmf.2/includes/user/User.php(2598)
#0 /srv/mediawiki/php-1.41.0-wmf.2/includes/libs/rdbms/database/Database.php(2610): User->{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#1 /srv/mediawiki/php-1.41.0-wmf.2/includes/libs/rdbms/database/DBConnRef.php(119): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#2 /srv/mediawiki/php-1.41.0-wmf.2/includes/libs/rdbms/database/DBConnRef.php(680): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#3 /srv/mediawiki/php-1.41.0-wmf.2/includes/user/User.php(2610): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure)
#4 /srv/mediawiki/php-1.41.0-wmf.2/extensions/LdapAuthentication/includes/LdapAuthenticationPlugin.php(1261): User->saveSettings()
#5 /srv/mediawiki/php-1.41.0-wmf.2/extensions/LdapAuthentication/includes/LdapPrimaryAuthenticationProvider.php(133): LdapAuthenticationPlugin->updateUser(User)
#6 /srv/mediawiki/php-1.41.0-wmf.2/includes/HookContainer/HookContainer.php(338): LdapPrimaryAuthenticationProvider->onUserLoggedIn(User)
#7 /srv/mediawiki/php-1.41.0-wmf.2/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#8 /srv/mediawiki/php-1.41.0-wmf.2/includes/HookContainer/HookRunner.php(4335): MediaWiki\HookContainer\HookContainer->run(string, array)
#9 /srv/mediawiki/php-1.41.0-wmf.2/includes/auth/AuthManager.php(2573): MediaWiki\HookContainer\HookRunner->onUserLoggedIn(User)
#10 /srv/mediawiki/php-1.41.0-wmf.2/includes/auth/AuthManager.php(793): MediaWiki\Auth\AuthManager->setSessionDataForUser(User, boolean)
#11 /srv/mediawiki/php-1.41.0-wmf.2/includes/auth/AuthManager.php(462): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#12 /srv/mediawiki/php-1.41.0-wmf.2/includes/specialpage/AuthManagerSpecialPage.php(374): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#13 /srv/mediawiki/php-1.41.0-wmf.2/includes/specialpage/AuthManagerSpecialPage.php(506): AuthManagerSpecialPage->performAuthenticationStep(string, array)
#14 /srv/mediawiki/php-1.41.0-wmf.2/includes/htmlform/HTMLForm.php(744): AuthManagerSpecialPage->handleFormSubmit(array, VFormHTMLForm)
#15 /srv/mediawiki/php-1.41.0-wmf.2/includes/specialpage/AuthManagerSpecialPage.php(437): HTMLForm->trySubmit()
#16 /srv/mediawiki/php-1.41.0-wmf.2/includes/specialpage/LoginSignupSpecialPage.php(323): AuthManagerSpecialPage->trySubmit()
#17 /srv/mediawiki/php-1.41.0-wmf.2/includes/specialpage/SpecialPage.php(701): LoginSignupSpecialPage->execute(NULL)
#18 /srv/mediawiki/php-1.41.0-wmf.2/includes/specialpage/SpecialPageFactory.php(1491): SpecialPage->run(NULL)
#19 /srv/mediawiki/php-1.41.0-wmf.2/includes/MediaWiki.php(328): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#20 /srv/mediawiki/php-1.41.0-wmf.2/includes/MediaWiki.php(926): MediaWiki->performRequest()
#21 /srv/mediawiki/php-1.41.0-wmf.2/includes/MediaWiki.php(579): MediaWiki->main()
#22 /srv/mediawiki/php-1.41.0-wmf.2/index.php(50): MediaWiki->run()
#23 /srv/mediawiki/php-1.41.0-wmf.2/index.php(46): wfIndexMain()
#24 /srv/mediawiki/w/index.php(3): require(string)
#25 {main}

Impact

Notes

Details

Request URL: https://wikitech.wikimedia.org/w/index.php?returnto=*&returntoquery=*&title=*

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	PRODUCTION ERROR	Krinkle	T334232 Wikitech experiencing a spike of stale cache errors since 2023-03-15
		Resolved	PRODUCTION ERROR	None	T333925 Error during MFA setup for wikitech.wikimedia.org: MWException: CAS update failed on user_touched. The version of the user to be saved is older than the current version.

Event Timeline

• WMDE_Norman created this task.Apr 4 2023, 9:49 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2023, 9:49 AM

This might be T249623

Restricted Application added a project: Wikimedia-production-error. · View Herald TranscriptApr 4 2023, 10:02 AM

Aklapper renamed this task from Error during MFA setup for wikitech.wikimedia.org to Error during MFA setup for wikitech.wikimedia.org: MWException: CAS update failed on user_touched. The version of the user to be saved is older than the current version..Apr 4 2023, 10:04 AM

Aklapper set Request URL to https://wikitech.wikimedia.org/w/index.php?returnto=*&returntoquery=*&title=*.

Aklapper updated the task description. (Show Details)

FWIW I confirm this is a legitimate request.
After the exception @WMDE_Norman is unable to log in to Wikitech - WMDE would appreciate help with getting the access back, assuming the issue blocking the access has been resolved. Thank you.

This error really looks like a transient failure related to a mismatch of database and cache state. The error comes from https://gerrit.wikimedia.org/g/mediawiki/core/+/884efec41c352bb21b82e2aabaf15d3f6f7b6f3b/includes/user/User.php#2598. I think generally this ticket should merge into T249623: Logging in on Wikitech can throw fatal from LdapAuthentication "CAS update failed on user_touched".

In T333925#8753894, @WMDE-leszek wrote:

FWIW I confirm this is a legitimate request.
After the exception @WMDE_Norman is unable to log in to Wikitech - WMDE would appreciate help with getting the access back, assuming the issue blocking the access has been resolved. Thank you.

What is @WMDE_Norman's Developer account name (wikitech user name) and what error are they now seeing when attempting to login to Wikitech? Did the 2FA setup apply to the database without giving them the needed local information to setup their TOTP client?

Wikitech user name: Norman Schwirz
He can confirm tomorrow but per the description having entered username and password he gets the MWException error page. He indeed didn't get the 2FA information.
Merging the error report into the known issue ticket sounds alright but Norman should get his access back too.

In T333925#8755388, @WMDE-leszek wrote:

Wikitech user name: Norman Schwirz

wikiadmin2023@10.64.32.114(labswiki)> select user_id, user_name from user where user_name='Norman Schwirz';
+---------+----------------+
| user_id | user_name      |
+---------+----------------+
|   35763 | Norman Schwirz |
+---------+----------------+
1 row in set (0.001 sec)

wikiadmin2023@10.64.32.114(labswiki)> select * from oathauth_users where id=35763;
Empty set (0.000 sec)

OATH 2FA is not active for https://wikitech.wikimedia.org/wiki/User:Norman_Schwirz.

He can confirm tomorrow but per the description having entered username and password he gets the MWException error page. He indeed didn't get the 2FA information.

The cache issue should be transient. The code block that emits the "CAS update failed on user_touched." error first invalidates the WANCache storage for the active user record. Assuming that there is not a network partition keeping that cache invalidation signal from reaching the cache service, the next page load for the same user should result in a cache miss and fresh data being pulled from the database.

Merging the error report into the known issue ticket sounds alright but Norman should get his access back too.

Agreed. Let's wait to hear what happens the next time that @WMDE_Norman tries to login to https://wikitech.wikimedia.org/ before deciding to merge or close this report.

Hello,

you assumed right my (user-)name is Norman Schwirz.
I can login again. My preferences page says: Two-factor authentication: None enabled and shows me an manage-button.
Should I try to setup Two-factor authentication again?

In T333925#8758679, @WMDE_Norman wrote:

Should I try to setup Two-factor authentication again?

Please do. Hopefully you will not suffer the same stale cache issue on the next attempt.

We do have a very similar report of stale cache at T334102: Wikitech: Preferences not updating after email change which seems to be repeatable. If you get the same CAS failure message again, please do report it back here so we can decide to spend more time trying to track down why things are getting out of sync.

Restricted Application added a project: User-bd808. · View Herald TranscriptApr 5 2023, 4:07 PM

As requested above I tried it again:

Login as Norman Schwirz
go to the preferences page
choose TOTP (because it seems to be the recommended one)
got the knows error:

Again I got the

Internal error
[96dcf57e-c4d8-4a01-9d3f-6c8bb99a8e76] 2023-04-06 07:14:49: Fatal exception of type "MWException"
{F36940912}

After reloading the website and login again I got a 500 internal error with this message:
[d25cba0d-36a3-4537-ab2d-a67b9a167d6c] 2023-04-06 07:34:02: Fatal exception of type "MWException"

Trying it in a new (private session) browser window does not work too:

I am locked out again.

In T333925#8761574, @WMDE_Norman wrote:

As requested above I tried it again:

Login as Norman Schwirz

go to the preferences page

choose TOTP (because it seems to be the recommended one)

got the knows error:

Again I got the

Internal error
[96dcf57e-c4d8-4a01-9d3f-6c8bb99a8e76] 2023-04-06 07:14:49: Fatal exception of type "MWException"
{F36940912}

After reloading the website and login again I got a 500 internal error with this message:
[d25cba0d-36a3-4537-ab2d-a67b9a167d6c] 2023-04-06 07:34:02: Fatal exception of type "MWException"

Trying it in a new (private session) browser window does not work too:

I am locked out again.

The fundamental error logged for this is "CAS update failed on user_touched for user ID '35763' (replica read)".

bd808 removed bd808 as the assignee of this task.Apr 6 2023, 3:02 PM

bd808 mentioned this in T334232: Wikitech experiencing a spike of stale cache errors since 2023-03-15.Apr 6 2023, 3:14 PM

bd808 added a parent task: T334232: Wikitech experiencing a spike of stale cache errors since 2023-03-15.Apr 6 2023, 3:28 PM

thcipriani moved this task from Untriaged to Apr 2023 on the Wikimedia-production-error board.Apr 13 2023, 3:48 PM

@WMDE_Norman We think that the config change from https://gerrit.wikimedia.org/r/912421 will have fixed the underlying issue that was causing the "CAS update failed" errors. This has been deployed, so when you have a chance, please do try to enable your 2FA protections on Wikitech again.

@WMDE_Norman Did you find time to check this again, per last comment? Thanks in advance

In T333925#8849335, @Aklapper wrote:

@WMDE_Norman Did you find time to check this again, per last comment? Thanks in advance

No reply; assuming this is resolved. If you still face this problem when using 2FA on wikitech, please reopen and elaborate. Thanks!

	F36940940: MWException2.png
	Apr 6 2023, 7:48 AM

Error during MFA setup for wikitech.wikimedia.org: MWException: CAS update failed on user_touched. The version of the user to be saved is older than the current version.Closed, ResolvedPublicPRODUCTION ERRORActions