Just remembered @Tgr deployed some OAuth stuff during the latest backport window - any thoughts?

Yeah, probably caused by rEOAUa750632f6b0e: Set 'sub' JWT field in client credentials access tokens.

T417820: TypeError: MediaWiki\Extension\OAuth\Entity\AccessTokenEntity::setUserIdentifier(): Argument #1 ($identifier) must be of type string, null given, called in AccessTokenEntity.php broke client credentials and that caused some 100/m errors; this is more like 1/m so maybe it's only partially broken? Could just be the luck of what bot is active, though.

(Also that broke getting an access token with client credentials, and this probably broke using an access token that was obtained with client credentials, so maybe one causes bots to retry more aggressively than the other.)

This is the snippet that sets ResourceServer::$user:

$userId = $request->getAttribute( 'oauth_user_id', 0 );
if ( !$userId ) {
	// Set anon user when no user id is present in the AT (machine grant)
	$this->user = User::newFromId( 0 );
	return;
}

$this->user = Utils::getLocalUserFromCentralId( $userId );

so apparently the value in sub is not empty but also not a valid ID. I guess the handling of the mw: prefix fails somehow? In general, we should make sure sub parsing fails in a safe way.

(Also we need to fix T418720: Logstash entries should include the application ID used for the request, it's hard to debug errors without knowing the details of the app that's causing them.)

Change #1251987 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/OAuth@master] Fix client credentials access tokens

https://gerrit.wikimedia.org/r/1251987

Change #1251987 merged by jenkins-bot:

[mediawiki/extensions/OAuth@master] Fix client credentials access tokens

https://gerrit.wikimedia.org/r/1251987

Change #1253623 had a related patch set uploaded (by Bartosz Dziewoński; author: Gergő Tisza):

[mediawiki/extensions/OAuth@wmf/1.46.0-wmf.19] Fix client credentials access tokens

https://gerrit.wikimedia.org/r/1253623

Change #1253623 merged by jenkins-bot:

[mediawiki/extensions/OAuth@wmf/1.46.0-wmf.19] Fix client credentials access tokens

https://gerrit.wikimedia.org/r/1253623

Mentioned in SAL (#wikimedia-operations) [2026-03-16T20:57:26Z] <catrope@deploy2002> Started scap sync-world: Backport for [[gerrit:1253623|Fix client credentials access tokens (T417278 T419921)]], [[gerrit:1253625|Enable $wgTrackMediaRequestProvenance on testwikis and beta cluster (T414338)]], [[gerrit:1253626|Configure $wgApiClientErrorSampleRate (T418957)]]

Mentioned in SAL (#wikimedia-operations) [2026-03-16T20:59:17Z] <catrope@deploy2002> matmarex, catrope: Backport for [[gerrit:1253623|Fix client credentials access tokens (T417278 T419921)]], [[gerrit:1253625|Enable $wgTrackMediaRequestProvenance on testwikis and beta cluster (T414338)]], [[gerrit:1253626|Configure $wgApiClientErrorSampleRate (T418957)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-03-16T21:05:37Z] <catrope@deploy2002> Finished scap sync-world: Backport for [[gerrit:1253623|Fix client credentials access tokens (T417278 T419921)]], [[gerrit:1253625|Enable $wgTrackMediaRequestProvenance on testwikis and beta cluster (T414338)]], [[gerrit:1253626|Configure $wgApiClientErrorSampleRate (T418957)]] (duration: 08m 06s)