GlobalWatchlist: cross-wiki request to mediawiki.org or test.wikipedia.org failing with `badtoken`
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	DannyS712
	Nov 7 2023, 7:18 PM

Description

Error

You need to have mediawiki.org as one of the sites in your global watchlist. Since nothing has changed in the global watchlist code in weeks, I believe that this indicates something has changed with the API

message

An API error occurred while interacting with "www.mediawiki.org":
⧼badtoken⧽

trace

Debug.js:62 Error: GlobalWatchlistError: www.mediawiki.org:API.actuallyGetWatchlist #1 badtoken
    at Debug.js.GlobalWatchlistDebugger.error (Debug.js:56:18)
    at SiteBase.js.GlobalWatchlistSiteBase.error (SiteBase.js:84:19)
    at eval (SiteBase.js:107:9)
    at mightThrow (jquery.js:3489:29)
    at process (jquery.js:3557:12)

Impact

Changes cannot be fetched for mediawiki.org

Notes

Thankfully global watchlist has some client-side JS logging. mw.globalwatchlist.debug.debugLog.filter( ( m ) => m.includes( 'mediawiki.org:' ) ) shows for me:

"7: www.mediawiki.org:API.actuallyGetWatchlist #1 (called), with func & content:	["get",{"action":"query","formatversion":2,"list":"watchlist","wllimit":"max","wlprop":"ids|title|flags|loginfo|parsedcomment|timestamp|user|tags|expiry","wlshow":"unread","wltype":"edit|new|log","wlallrev":true}]"
"92: ERROR: www.mediawiki.org:API.actuallyGetWatchlist #1 badtoken	{"error":{"code":"badtoken","info":"⧼badtoken⧽","docref":"See https://www.mediawiki.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes."},"servedby":"mw-api-ext.codfw.main-586d55f64d-hn5l5"}"
"93: www.mediawiki.org:getWatchlist - error"

The same problem happens with test.wikipedia.org.
See also wikitech-l.

Details

Request URL: https://meta.wikimedia.org/wiki/Special:GlobalWatchlist

	Subject	Repo	Branch	Lines +/-
	Fix centralauthtoken key schema migration	mediawiki/extensions/CentralAuth	master	+1 -4
	Fix centralauthtoken key schema migration	mediawiki/extensions/CentralAuth	wmf/1.42.0-wmf.4	+1 -4

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	jnuche	T350080 1.42.0-wmf.4 deployment blockers
		Resolved	PRODUCTION ERROR	Tgr	T350723 GlobalWatchlist: cross-wiki request to mediawiki.org or test.wikipedia.org failing with `badtoken`

Event Timeline

DannyS712 triaged this task as Unbreak Now! priority.Nov 7 2023, 7:18 PM

DannyS712 created this task.

Restricted Application added a project: User-DannyS712. · View Herald TranscriptNov 7 2023, 7:18 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

IKhitron renamed this task from GlobalWatchlist: cross-wiki request to mediawiki.org failing with `badtoken` to GlobalWatchlist: cross-wiki request to mediawiki.org or test.wikipedia.org failing with `badtoken`.Nov 7 2023, 7:22 PM

IKhitron awarded a token.

IKhitron updated the task description. (Show Details)

IKhitron subscribed.

testwiki also has GW deployed, and there (from 1.42.0-wmf.4) the api works to retrieve changes from both mediawiki.org and testwiki, but on meta when I add testwiki to the global watchlist (or presumably any other 1.42.0-wmf.4 wiki) it fails, so the issue is api requests from wmf.3 to wmf.4

IKhitron updated the task description. (Show Details)Nov 7 2023, 7:25 PM

IKhitron added a subscriber: Urbanecm_WMF.Nov 7 2023, 7:31 PM

@Tgr said he'll look into it.

For what it's worth, I can reproduce this with test.wikipedia.org, but not with www.mediawiki.org.

Minimal code to reproduce is this, executed in the browser console:

await new mw.ForeignApi('https://test.wikipedia.org/w/api.php').get( { meta: 'userinfo' } )

Expected: Object { batchcomplete: "", query: {…} } (or something like this, depending on the browser)
Actual: Uncaught (in promise) badtoken (or something like this, depending on the browser)

The faulty error message "⧼badtoken⧽" is a separate problem, https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/972455 will fix that. (It seems this has always been broken, but no one has hit this code path before in sufficient numbers to notice.)

matmarex moved this task from Inbox, needs triage to Current Sprint on the MediaWiki-Platform-Team board.Nov 7 2023, 8:01 PM

Probably caused by rECAU39f704a0a19b: Begin migrating key formats for cache key lookups or writes (p.1) / T347223: Exception: Key contains invalid characters: centralauth:central-login-complete-token:1�À§À¢%2527%2522.

Token generation seems to work as expected:

tgr@mwmaint1002:~$ mwscript shell.php metawiki
> sudo $tokenStore = MediaWiki\Session\SessionManager::getGlobalSession()->getProvider()->sessionManager->getTokenStore()
> $token = '...'
> # when generated on meta
> $tokenStore->getMulti( [ "centralauth:api-token:$token", "global:api-token:centralauth:$token" ] )
= [
    "centralauth:api-token:..." => [...],
  ] 
> # when generated on mediawiki.org
> $tokenStore->getMulti( [ "centralauth:api-token:$token", "global:api-token:centralauth:$token" ] )
= [
    "centralauth:api-token:..." => [...],
    "global:api-token:centralauth:..." => [...],
  ]

Change 972464 had a related patch set uploaded (by Bartosz Dziewoński; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@master] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972464

gerritbot added a project: Patch-For-Review.Nov 7 2023, 8:37 PM

Change 972464 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972464

Change 972262 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@wmf/1.42.0-wmf.4] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972262

Change 972262 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.42.0-wmf.4] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972262

Mentioned in SAL (#wikimedia-operations) [2023-11-07T21:45:03Z] <tgr@deploy2002> Started scap: Backport for [[gerrit:972262|Fix centralauthtoken key schema migration (T347223 T350723)]]

Mentioned in SAL (#wikimedia-operations) [2023-11-07T21:46:23Z] <tgr@deploy2002> tgr: Backport for [[gerrit:972262|Fix centralauthtoken key schema migration (T347223 T350723)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-11-07T21:58:20Z] <tgr@deploy2002> Finished scap: Backport for [[gerrit:972262|Fix centralauthtoken key schema migration (T347223 T350723)]] (duration: 13m 17s)

Verified via Special:ApiSandbox that centralauthtokens work between meta and mediawiki.org.

Thanks - GlobalWatchlist is working properly too

DannyS712 moved this task from Unsorted to Reports on the User-DannyS712 board.Nov 7 2023, 9:59 PM

Confirming it started to work in last five minutes, and I get mediawiki.org results from timestamp after the problem started.

DannyS712 updated the task description. (Show Details)Nov 7 2023, 9:59 PM

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.5; 2023-11-14).Nov 7 2023, 10:00 PM

For history purposes.

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2023, 10:10 PM

DAlangi_WMF mentioned this in T350081: 1.42.0-wmf.5 deployment blockers.Nov 13 2023, 9:55 AM

	F41468064: CGI6pI0.jpeg.jpg
	Nov 7 2023, 10:02 PM

GlobalWatchlist: cross-wiki request to mediawiki.org or test.wikipedia.org failing with `badtoken`Closed, ResolvedPublicPRODUCTION ERRORActions