Wikimedia\NormalizedException\NormalizedException: Authentication failed because of inconsistent provider array
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	Reedy
	Mar 5 2025, 11:08 PM

Project Tags

Referenced Files

	F58620272: Screenshot Capture - 2025-03-06 - 16-48-48.png
	Mar 6 2025, 8:15 PM

Subscribers

Aklapper

Reedy

Tgr

Description

Error

mwversion: 1.44.0-wmf.19
reqId: 1e1f2ab2-8a69-4fc3-8695-e4470eea6964
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   Wikimedia\NormalizedException\NormalizedException: Authentication failed because of inconsistent provider array

Frame	Location	Call
from	/srv/mediawiki/php-1.44.0-wmf.19/includes/auth/AuthManager.php(519)
#0	/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(401)	MediaWiki\Auth\AuthManager->continueAuthentication(array)
#1	/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(533)	MediaWiki\SpecialPage\AuthManagerSpecialPage->performAuthenticationStep(string, array)
#2	/srv/mediawiki/php-1.44.0-wmf.19/includes/htmlform/HTMLForm.php(824)	MediaWiki\SpecialPage\AuthManagerSpecialPage->handleFormSubmit(array, MediaWiki\HTMLForm\CodexHTMLForm)
#3	/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/AuthManagerSpecialPage.php(464)	MediaWiki\HTMLForm\HTMLForm->trySubmit()
#4	/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/LoginSignupSpecialPage.php(404)	MediaWiki\SpecialPage\AuthManagerSpecialPage->trySubmit()
#5	/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/SpecialPage.php(729)	MediaWiki\SpecialPage\LoginSignupSpecialPage->execute(null)
#6	/srv/mediawiki/php-1.44.0-wmf.19/includes/specialpage/SpecialPageFactory.php(1737)	MediaWiki\SpecialPage\SpecialPage->run(null)
#7	/srv/mediawiki/php-1.44.0-wmf.19/includes/actions/ActionEntryPoint.php(503)	MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
#8	/srv/mediawiki/php-1.44.0-wmf.19/includes/actions/ActionEntryPoint.php(145)	MediaWiki\Actions\ActionEntryPoint->performRequest()
#9	/srv/mediawiki/php-1.44.0-wmf.19/includes/MediaWikiEntryPoint.php(202)	MediaWiki\Actions\ActionEntryPoint->execute()
#10	/srv/mediawiki/php-1.44.0-wmf.19/index.php(58)	MediaWiki\MediaWikiEntryPoint->run()
#11	/srv/mediawiki/w/index.php(3)	require(string)
#12	{main}

Impact

Notes

Details

Request URL: https://www.mediawiki.org/w/index.php?campaign=*&centralauthLoginToken=*&title=*

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
Simplify return URL parameter handling	Gergő Tisza	mediawiki/extensions/CentralAuth	master	+2 -4
SUL3: Attach SUL mode to the return URL of local wiki	Gergő Tisza	mediawiki/extensions/CentralAuth	wmf/1.44.0-wmf.19	+6 -0
SUL3: Attach SUL mode to the return URL of local wiki	D3r1ck01	mediawiki/extensions/CentralAuth	master	+6 -0

Customize query in gerrit

Related Objects

Mentioned In: T392233: Wikimedia\NormalizedException\NormalizedException: Authentication failed because of inconsistent provider array
T388252: PHP Deprecated: str_ends_with(): Passing null to parameter #1 ($haystack) of type string is deprecated
T388177: Wikimedia\NormalizedException\NormalizedException: User ID: {centralUserId} mismatch with {storedUserId} for user: {username}
T384007: SUL3 Phase 1: All new account creation on group 0 and group 1 wikis
Mentioned Here: T388177: Wikimedia\NormalizedException\NormalizedException: User ID: {centralUserId} mismatch with {storedUserId} for user: {username}
T388252: PHP Deprecated: str_ends_with(): Passing null to parameter #1 ($haystack) of type string is deprecated

Event Timeline

Reedy created this task.Mar 5 2025, 11:08 PM

Restricted Application added a project: MediaWiki-Core-Platform-Team. · View Herald TranscriptMar 5 2025, 11:08 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Tgr mentioned this in T384007: SUL3 Phase 1: All new account creation on group 0 and group 1 wikis.Mar 6 2025, 4:27 PM

This is a security check introduced during the SUL3 project to make sure authentication configuration cannot change mid-authentication (the likely cause would be some bug in how we determine whether a user is opted into SUL3, but it could also be caused by a deployment changing auth config while someone is going through login/signup).

There have been two small spikes (logstash):

Screenshot Capture - 2025-03-06 - 16-48-48.png (1,352×590 px, 48 KB)

all on group 0 wikis during SUL3 login, I think? The logs are a bit confusing - server is never auth.wikimedia.org (that's from $_SERVER['SERVER_NAME'] which I think should be the real domain, or maybe that gets overriden by the Apache virtual host name?), referrer is https://auth.wikimedia.org/, url is /w/index.php?centralauthLoginToken=<token>&title=Special:UserLogin which I can't quite place (if it's on the auth domain, why doesn't it have usesul3=1? if it's during return, why doesn't it have /return in the special page name?)

The old provider array is

preauth: U...(private), CentralAuthSharedDomainPreAuthenticationProvider, AccountCreationEventProvider
primaryauth: CentralAuthTemporaryPasswordPrimaryAuthenticationProvider, CentralAuthRedirectingPrimaryAuthenticationProvider, CentralAuthPrimaryAuthenticationProvider
secondaryauth: CampaignsSecondaryAuthenticationProvider, ResetPasswordSecondaryAuthenticationProvider, EmailNotificationSecondaryAuthenticationProvider

The new provider array is

preauth: ThrottlePreAuthenticationProvider, G...(private), U...(private), SpamBlacklistPreAuthenticationProvider, TitleBlacklistPreAuthenticationProvider, AntiSpoofPreAuthenticationProvider, CentralAuthSharedDomainPreAuthenticationProvider, AccountCreationEventProvider, IPReputation\PreAuthenticationProvider, AbuseFilterPreAuthenticationProvider, CaptchaPreAuthenticationProvider
primaryauth: CentralAuthTemporaryPasswordPrimaryAuthenticationProvider, CentralAuthRedirectingPrimaryAuthenticationProvider, CentralAuthPrimaryAuthenticationProvider, LocalPasswordPrimaryAuthenticationProvider
secondaryauth CheckBlocksSecondaryAuthenticationProvider, CentralAuthSecondaryAuthenticationProvider, CampaignsSecondaryAuthenticationProvider OATHAuth\\SecondaryAuthenticationProvider, ResetPasswordSecondaryAuthenticationProvider, EmailNotificationSecondaryAuthenticationProvider

which looks like SUL3 flipping from enabled to disabled mid-login (on the local domain). We are supposed to have explicit usesul3 query flags to prevent that, maybe there's something wrong with that?
(Also I think U...(private) should be disabled for SUL3 login on the local domain.)

Reedy moved this task from Untriaged to Mar 2025 on the Wikimedia-production-error board.Mar 6 2025, 10:28 PM

I think it happens when the user's IP changes mid-authentication, and they take more than 5 minutes so the usesul3 cookie expires. I confirmed that we don't set usesul3=1 in the return URL, we should fix that.

Tgr added a project: SUL3.Mar 6 2025, 11:06 PM

Change #1125496 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@master] SUL3: Attach SUL mode to the return URL of local wiki

https://gerrit.wikimedia.org/r/1125496

gerritbot added a project: Patch-For-Review.Mar 7 2025, 6:28 PM

I guess I just assumed AuthPreserveQueryParams would handle this (since we do call it when generating the return URL) but then there is no reason for the usesul3 query parameter to be set on the initial request to the login page, where we are generating the return URL.

Change #1125496 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] SUL3: Attach SUL mode to the return URL of local wiki

https://gerrit.wikimedia.org/r/1125496

Maintenance_bot removed a project: Patch-For-Review.Mar 8 2025, 9:30 PM

• matmarex moved this task from Inbox, needs triage to In progress (DO NOT USE) on the MediaWiki-Core-Platform-Team board.Mar 8 2025, 9:34 PM

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.20; 2025-03-11).Mar 8 2025, 10:00 PM

Change #1125643 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@master] Simplify return URL parameter handling

https://gerrit.wikimedia.org/r/1125643

gerritbot added a project: Patch-For-Review.Mar 9 2025, 6:39 PM

Tgr closed this task as Resolved.Mar 9 2025, 7:03 PM

Tgr assigned this task to • DAlangi_WMF.

Change #1126049 had a related patch set uploaded (by Gergő Tisza; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.19] SUL3: Attach SUL mode to the return URL of local wiki

https://gerrit.wikimedia.org/r/1126049

Change #1126049 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.19] SUL3: Attach SUL mode to the return URL of local wiki

https://gerrit.wikimedia.org/r/1126049

Mentioned in SAL (#wikimedia-operations) [2025-03-10T14:49:53Z] <tgr@deploy2002> tgr: Backport for [[gerrit:1126050|SpecialCentralAutoLogin: Handle nullable wiki ID (T388252)]], [[gerrit:1126049|SUL3: Attach SUL mode to the return URL of local wiki (T388067)]], [[gerrit:1126051|Log and add user IDs that mismatch in the runtime exception (T388177)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot mentioned this in T388252: PHP Deprecated: str_ends_with(): Passing null to parameter #1 ($haystack) of type string is deprecated.Mar 10 2025, 2:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-10T14:58:12Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1126050|SpecialCentralAutoLogin: Handle nullable wiki ID (T388252)]], [[gerrit:1126049|SUL3: Attach SUL mode to the return URL of local wiki (T388067)]], [[gerrit:1126051|Log and add user IDs that mismatch in the runtime exception (T388177)]] (duration: 15m 48s)