This really sounds like an ask for deployment tool functionality (something like scap3) in a tool that is a service control instead

context: https://wikis.world/@LucasWerkmeister/116400150522373087

Seems to be fixed now, thanks!

Could be, yeah. I guess we could add a first version that only supports application/x-www-form-urlencoded, and that should support the majority of use cases (all of them?), as long as users don’t accidentally send multipart/form-data instead.

Sounds reasonable to me, thanks. (If I hear any other complaints from ACDC users I’ll report it here and/or there as seems appropriate.)

Sounds good, thanks. (FWIW, I haven’t heard any new complaints similar to T421161, but I have very limited visibility here – the badtoken API errors in Grafana are probably the best data we have.)

Now it’s working without Scribunto, thanks @Fomafix!

I don’t see how that’s a reason to close the task? I don’t really care whether you think it’s related to the DC switchover or not, I care about the issue getting fixed. And to me the Grafana graph linked in the task description, looking at badtoken errors in the last 30 days, doesn’t really look like a return to normal yet (though the logarithmic scale makes it hard to interpret, and AFAICT my volunteer account doesn’t have sufficient privileges to preview an edited version with a linear scale):

Now (following T419034#11788469) it works! This edit was made with just the client ID (no client secret) and successfully refreshed the access token before making the edit. (The 401 response even had CORS headers, including access-control-expose-headers: Retry-After,WWW-Authenticate, so that this worked fully in the browser.)

I’ve now updated m3api-oauth2 to handle non-MediaWiki HTTP-level errors with WWW-Authenticate response headers, in m3api v1.1.0 + m3api-oauth2 v1.0.4 (also v1.0.5). As far as I can tell, this was successful – this edit automatically refreshed the access token (I added some console.log()s in node_modules/ to make it visible).

See also: T410883: Support HTTP QUERY method as standard alternative to Promise-Non-Write-API-Action header

Still happening, still causing accidental edits: https://meta.wikimedia.org/w/index.php?title=Talk:QuickCategories&diff=prev&oldid=30348897

This change also broke Tool-quickcategories:

In T373871#11208344, @HCoplin-WMF wrote:

Updating as low priority since we don't think anyone is actually using it right now

I don’t know how to answer that question beyond the IRC log link that’s already in the task description… there was a suspicious increase in errors right around the time of the DC switch, and the switchover announcement said to file a task for any issues, so I did.

(This task isn’t related to confidential vs. non-confidential or browser-side vs. non-CORS-limited clients though, it affects everyone as far as I’m aware.)

Seems to work on Beta and individual consumers) \o/

In T421552#11760041, @JeanFred wrote:

At startup time, the Flask app imports a module which imports pywikibot ; but I would have thought only actual requests would trigger a pywikibot.Site initialization. Yet the logs seem to indicate that straight off the bat pywikibot does some throttling...

Hm, apparently this is… intentional? Add handler for /?url=... testing, from 2019, does what it says on the tin.

In T420833#11752055, @bd808 wrote:

We need the https://wikitech.wikimedia.org/wiki/Requestctl stack in Beta.

ACDC is a Commons gadget. (It doesn’t have a Phabricator tag yet, though maybe I should apply one. Not sure yet.)

This might be caused by a general issue due to the datacenter switchover: T421168: Session store issues causing badtoken errors, session failures, logouts (late March–April 2026)

I don’t know what’s going on here but it seems worth investigating, so filing the task for visibility :)

ACDC uses mw.Api.postWithEditToken:

In T420837#11737941, @Count_Count wrote:

Apparently the cdnjs auto-update has been broken for a while, see https://github.com/cdnjs/cdnjs/issues/14263#issuecomment-3800585676

Should be fixed now, thanks for the report! The perils of having tools translated into more than 50 languages :)

The Moroccan Arabic templates were merged and deployed ca. ten minutes ago :)

I would probably go for the history rewrite – mainly for those cache directories which otherwise bloat the history forever. (Although the toolforge build service clones with --depth=1 if I’m not mistaken, so I guess it wouldn’t affect that anymore. But it would still affect any developer who clones the repo.) And at that point you might as well drop the credentials too. But if you want to keep the history, I think it would probably be fine to do so at this point, now that the database credentials have been refreshed and the kubernetes certificates will expire soon; you should just wait a few more days, if I’m not mistaken:

Great, thanks! I guess this means the k8s certs were already rotated automatically?

(Tagging Toolforge in the hope of finding someone who can do the credential rotation. Last I checked, I think I didn’t have enough access to do it myself – I have root on the Toolforge servers, but I don’t have cloudcontrolXXX access for to regenerate replica.my.cnf. I might be able to regenerate kubernetes credentials for tools but I’m not sure I want to risk that without being familiar with the process, tbh.)

It might be okay to make the task public, but I think it needs to stay open – IIUC, a Toolforge admin (or Cloud VPS admin?) needs to rotate the credentials (k8s certificate, ToolsDB password), since we don’t know who else might have accessed them.

In T419034#11675904, @Tgr wrote:

I think the relevant spec is RfC 6750:
…
No idea whether we actually implement that (either in Envoy or in MediaWiki).

Side note: I tried to see if there’s a standard way to signal expired access tokens in a response, but it doesn’t sound like it; RFC 6749 intentionally omits this:

In T419034#11675121, @matmarex wrote:

In the client-side example, this will result in a CORS error because the response does not have an Access-Control-Allow-Origin header. m3api’s code can’t even see the response.

Yeah, this is rough. I just filed T418969 about the same problem for the rate-limited requests. The solution I suggested there would probably work here as well (let through the preflight OPTIONS request, and add some headers to the GET/POST request's response)… it should be fine to let the preflight OPTIONS request happen, right?

In T419034#11674561, @Tgr wrote:

m3api-oauth2 can’t register a handler for this kind of error.

That might be a problem for 429 errors from Envoy in the future.

Probably not much, though we should at least remove this sentence once we’ve confirmed T323855 is fixed.

In T419034#11674545, @Tgr wrote:

So basically, Envoy just needs to pass through requests with expired access tokens to MediaWiki? The Access-Control-Allow-Origin issue is specific to Envoy, wouldn't be an issue with the MediaWiki error response, right?

In T419034#11674483, @Tgr wrote:

I'm not sure anything outside MediaWiki does JSON responses? Also the JWT is validated in three layers (Varnish, API Gateway, MediaWiki) but I think the first two would just apply anonymous-request-level rate limiting rather than reject the request, even if the JWT is expired. Not really sure about it though.

On the MediaWiki level, we did some big upgrades recently (T261462: Migrate OAuth extension back from wikimedia/oauth2-server fork to upstream) so that could also have caused a regression.

The error is indeed unrelated and affects both examples: ⇒ T419034: Custom OAuth 2 error from Wikimedia infrastructure breaks automatic retry of requests

Well, the refresh doesn’t quite work properly, though I’m not sure it’s due to this issue. My web app makes a meta=tokens request, and the API responds with {"httpCode":401,"httpReason":"Jwt is expired"}, but the response is missing Access-Control-Allow-Origin, so my code crashes before it can even try a refresh.

Seems to work! I made this edit with the following local diff to the webapp-clientside-vite-guestbook m3api example:

I guess T418580: Deploy 2FA requirement using $wgRestrictedGroups to Wikimedia production, instead of OATHAuth's custom config (tech news) will effectively resolve this task, removing the confusion by tying actual group membership to 2FA?

I’m guessing the Node buildpack is itself outdated?

Well, I just tried sending an email to myself via Special:EmailUser and it worked fine. So from my side this seems to have worked :)

In T417839#11631101, @matmarex wrote:

@LucasWerkmeister @Gerges @Rtconner I'd appreciate if you could test your applications as well (and thanks for the bug reports!).

In T417839#11630340, @matmarex wrote:

I am pretty sure it comes from Envoy, and not MediaWiki: https://github.com/envoyproxy/envoy/blob/6dc915a34553d1cb959908fd7f185ee73725286c/source/common/jwt/status.cc#L65

In T417839#11630289, @Jdforrester-WMF wrote:

Is this related to T417820: TypeError: MediaWiki\Extension\OAuth\Entity\AccessTokenEntity::setUserIdentifier(): Argument #1 ($identifier) must be of type string, null given, called in AccessTokenEntity.php, the other OAuth-related train-blocker, or are they different?

Boldly making this a train blocker (⇒ UBN!) for now; I don’t know how widely OAuth 2 is used compared to OAuth 1.0a (which doesn’t seem to be affected), but given that several users noticed the issue already (#wikimedia-cloud) I think it’s reasonable to guess that this is causing some breakage.

haven’t compared their histories yet, in part because Git’s “dubious ownership” security mitigation is annoying

There are a bunch of separate Git repositories in FNBot/ (along with more directories that aren’t Git repos and which will just go into the main request-legacycode repo):

I don’t remember anything ^^ but T389540#10774116 is probably most of what I found at the time (sadly with less “methodology” than I’d like now).

Thanks! I can try to help out later, but not sure I’ll be able to get to it this weekend (depends on how smoothly a certain MediaWiki upgrade will go ^^).

I’ve encountered the same restriction in this job (though admittedly I was just trying to confirm the Cloud VPS IP space, so it’s not like this is blocking any work from me at the moment).

(Acknowledged, but I just want to point out how preposterous it is that letting just one cloud provider access a notionally public website can be considered “unreasonable” in this day and age. Fuck scrapers :( )

Yup, thank you! Committed to the main repo here: https://gitlab.wikimedia.org/repos/m3api/m3api-oauth2/-/commit/31c2577983

I guess the way to test this would be to deploy the new build, test it in the matrix-telegram-test gateway, and be ready to quickly restore the previous image if it doesn’t work out? (Or use #wikimedia-cloud and let the people in there just deal with the test messages.)

In T414864#11538068, @dancy wrote:

In T414864#11532038, @LucasWerkmeister wrote:

FWIW, I’ve tried to get m3api-oauth2 CI running on the WMCS runners instead (wmcs tag), but so far haven’t managed to get Chrome/Chromium running there yet (latest job).

I was going to suggest using the wmcs runners. It's unclear why that job failed. Let me know if you want to team up on it interactively to debug.

(Ideally this would’ve been reported as a non-public Security task, but now that there’s a public merge request detailing the issue I’m not sure if it’s useful to still security-protect the issue.)

Previously: T286415: XSS in ISA tool

FWIW, I’ve tried to get m3api-oauth2 CI running on the WMCS runners instead (wmcs tag), but so far haven’t managed to get Chrome/Chromium running there yet (latest job).

According to this screenshot the IP address of that particular job runner was 159.203.90.138. Which is apparently in 159.203.0.0/16, somewhere in DigitalOcean, LLC.

Just to mention it here – the group mechanism seems to work like a charm, I just noticed that the new m3api-rest repo got picked up without any issue \o/

In T316002#8178715, @Asartea wrote:

I can reproduce this on en.wikipedia.org with https://en.wikipedia.org/wiki/User:Chlod/sandbox.js

AFAICT this is still happening. @Ladsgroup updated wdvd to PHP 8.4, but webservice status is reporting PHP 7.3, probably because that’s what in the public_html/service.template file.