Page MenuHomePhabricator

Authenticated cross-origin requests are being throttled as if unauthenticated (centralauth)
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  1. Disable third-party cookies
  2. Log out (not sure if this is actually necessary)
  3. Log back in at en.wikipedia.org,
  4. Make at least 1000 ForeignApi requests to en.wikitionary.org, e.g.:
let a = new mw.ForeignApi("https://en.wiktionary.org/w/api.php", {ajax:{headers:{'Api-User-Agent':"Rate Limit Test (User:MyUserName)"}}});
let r = () => ({action:"query",requestid:Math.random()});

for(let i = 0; i < 400; i++)
  console.log(i, await Promise.all([a.get(r()),a.get(r()),a.get(r())]))

What happens?:

Works for a while then:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://en.wiktionary.org/w/api.php?action=query&format=json&origin=https%3A%2F%2Fen.wikipedia.org&centralauthtoken=...&requestid=0.815467728280899. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 429.

What should have happened instead?:

All requests succeed. There are currently no limits on authenticated browser traffic, and any future limits will be much greater than 1000 anyway. I am using a browser, and centralauthtoken is a form of authentication.

This is contrived, of course, but see https://www.mediawiki.org/wiki/Talk:Wikimedia_APIs/Rate_limits#Script-using_humans_are_not_bots where @Tom.Reding apparently tripped this bug in course of normal work.

Event Timeline

Reedy renamed this task from Authenticated crtoss-origin requests are being throttled as if unauthenticated to Authenticated cross-origin requests are being throttled as if unauthenticated.Mar 16 2026, 9:43 PM
suffusion_of_yellow renamed this task from Authenticated cross-origin requests are being throttled as if unauthenticated to Authenticated crtoss-origin requests are being throttled as if unauthenticated.Mar 16 2026, 9:43 PM
suffusion_of_yellow renamed this task from Authenticated crtoss-origin requests are being throttled as if unauthenticated to Authenticated cross-origin requests are being throttled as if unauthenticated.
suffusion_of_yellow added subscribers: matmarex, Tgr.

Rate limits are enforced in layers (the edge layer and the gateway layer) that sit on top of mediawiki. They do not (and cannot efficiently) have access to the data store that MediaWiki uses to validate centralauth tokens. The tokens that are honored by the rate limiting systems are JWTs - tokens that contain all relevant information in themselves, and are cryptographically signed to prevent manipulation.

To make it possible for centralauth tokens to grant elevated limits, we would have to change how they are generate and processed, and they would become much larger. For them to be processed at the edge layer it would also be much better if they could be sent as a header rather than a query parameter.

This isn't impossible, but it's not trivial either. @matmarex what do you think? Could we change centralauthtoken so it returns a short lived JWT that could also be used as a bearer token or similar?

pmiazga subscribed.

Moving to Q3 Kanban board as this seems directly related to our recent API rate-limiting work

I think it can be done, I will look into it.

There are really two parts to the change:

  1. Changing the token from a short hex string (around 40 bytes) to a JWT (around 900 bytes). Once the token is generated, we can keep treating it as an opaque string without major changes to the code. This should make no difference for the users of the APIs (it's still pretty short), but we'll need to make sure the 20x increase doesn't cause some storage or performance problems on our side. These tokens are very short-lived though (1 minute), so there should never be too many of them stored in absolute terms.
  1. Teaching the API gateway where to find this token, or changing how they are used to one of the formats the gateway already knows. Currently, the tokens are passed:
    • For api.php requests, in the centralauthtoken={token} URL query parameter (they must be passed in the URL even for POST requests, so that they will be included in preflight OPTIONS requests too)
    • For rest.php requests, in the Authorization: CentralAuthToken {token} header

I hope parsing the header will be easy (it's just like a bearer token). Parsing the query parameter may be trickier, but IIRC you already had to add query parameter support for something else (I don't remember where I saw that), so I hope it's feasible as well. In theory we could change the api.php requests to accept the token from a header as well, but that would require significant code changes (and not just in our code, but probably some gadgets too), we could practically never remove the old way, and I'm reluctant to add yet another authentication method when the reason we're having so much trouble here is that we already have too many of them.

I'm probably missing something obvious here, but this seems overly complicated. If the token isn't valid, they'll just get a badtoken response back from the API, yes? So why would someone flood the API with thousands of requests, if they aren't getting anything useful back?

And if the token is valid, it must correspond to a action=centralauthtoken request from an account, and would have been throttled at the same level as any other action from that account.

@suffusion_of_yellow Ideally, we would like to be able to reject rate-limited requests without doing all the work that goes into looking up the token and generating a badtoken response.

You make a good point though. We probably could just say that any request with a centralauthtoken must have already passed the rate-limit checks when the token was generated, exempt those requests from rate limits, and rely on other layers of protection for the case of someone flooding the API with requests that won't return anything useful. We're adding similar exemptions for OPTIONS requests in T418957 and for certain API actions in T419130.

On the other hand, adding yet another different exemption may be more work than just changing the format of the token.


  1. Changing the token from a short hex string (around 40 bytes) to a JWT (around 900 bytes). (…) we'll need to make sure the 20x increase doesn't cause some storage or performance problems on our side. (…)

Daniel suggested that we could just put the short hex string we currently use in a custom field inside the JWT, and "unwrap" it before sending the token to the backend storage. That will ensure there will be no storage problems.

  1. Teaching the API gateway where to find this token (…) I hope parsing the header will be easy (it's just like a bearer token). Parsing the query parameter may be trickier (…)

I found where this is configured: https://gerrit.wikimedia.org/g/operations/deployment-charts/+/7f1601a5abb49f09f51f9fffcfac70e8aa961936/charts/api-gateway/templates/_config.yaml#227
And found the relevant documentation: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/jwt_authn_filter#jwtprovider
It looks like this is easy – we just add from_params: / - centralauthtoken and from_headers: / name: Authorization / value_prefix: CentralAuthToken and we're done.

And if the token is valid, it must correspond to a action=centralauthtoken request from an account, and would have been throttled at the same level as any other action from that account.

I was assuming that a centralauthtoken can be used to make many requests while it is valid. Is that not the case?

No, they're single-use.

No, they're single-use.

I see.

...do they have to be?

They should be, not for any technical reason, but because they are passed in the URL (they have to be to pass preflight requests), so they could easily leak (e.g. in logs somewhere or due to the user copy-pasting an error message including the URL); and since they allow full access to your account, they've been made single-use and short-lived to mitigate the risk.

They should be, not for any technical reason, but because they are passed in the URL (they have to be to pass preflight requests), so they could easily leak (e.g. in logs somewhere or due to the user copy-pasting an error message including the URL); and since they allow full access to your account, they've been made single-use and short-lived to mitigate the risk.

Yea, I get the need for them to be short-lived. My assumption was that they are valid for a minute or so, but would still allow you to make multiple requests during that time. But then you'd have to detect when they expire, get a fresh token, and retry your request. Getting a new token for every request is simpler...

I'm asking because generating a JWT is computationally expensive. Not terribly so, doing it 10 times per second is not nothing. We use RSA, right? We could consider using AES for these. Much faster, but adds operational risk.

Hm... maybe we could actually re-use the token but associate it with a counter? Idea:

  • We generate a JWT that is good for an hour or so
  • The JWT contains a budget key (the string we currently use as the token)
  • We store a counter (the budget) for that key.
  • Every time you use a centralauthtoken, that budget goes down by one.
  • If the budget is 0, the request fails.
  • Every time you ask for a centralauthtoken, that budget goes up by one.
    • If the JWT is expired (or about to expire) we generate a new one and start the budget at 1, and delete the old budget.
    • we'd need to store the JWT expiry timestamp along with the current budget

I'm asking because generating a JWT is computationally expensive. Not terribly so, doing it 10 times per second is not nothing. We use RSA, right? We could consider using AES for these. Much faster, but adds operational risk.

I did some testing in production, using the WikimediaDebug extension to profile some requests that generate a JWT (logging in with a bot password). The JWT generation takes <10ms on these requests (and that's with profiling enabled). Here's a representative example: https://performance.wikimedia.org/excimer/profile/103094dfd0d2889d (the JWT generation call is the tiny bar here: F73158105)

We only serve at most 20 action=centralauthtoken requests per second: https://grafana.wikimedia.org/d/000000559/mediawiki-action-api-breakdown?orgId=1&from=now-7d&to=now&timezone=utc&var-module=centralauthtoken so I am sure that the added load will not be significant for us. The added <10ms latency for the users should not be significant either.

I think we can just do the simple thing, but please double-check my numbers. Meanwhile, I'll start working on a patch.

Change #1255900 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/CentralAuth@master] Wrap 'centralauthtoken' in a JWT

https://gerrit.wikimedia.org/r/1255900

I think we can just do the simple thing, but please double-check my numbers. Meanwhile, I'll start working on a patch.

Yea, should be fine, thank you for checking!
May be worth putting a comment in the code that points here, so if there is a problem in the future, we can come pack to the ideas and analysis.

Change #1259242 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] rest gateway: add support for centralauthtoken

https://gerrit.wikimedia.org/r/1259242

Change #1261439 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/extensions/CentralAuth@master] Add ApiCentralAuthTokenTest

https://gerrit.wikimedia.org/r/1261439

Change #1255900 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Wrap 'centralauthtoken' in a JWT

https://gerrit.wikimedia.org/r/1255900

Change #1261439 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Add ApiCentralAuthTokenTest

https://gerrit.wikimedia.org/r/1261439

Change #1261545 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/CentralAuth@wmf/1.46.0-wmf.21] Wrap 'centralauthtoken' in a JWT

https://gerrit.wikimedia.org/r/1261545

Change #1261545 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.46.0-wmf.21] Wrap 'centralauthtoken' in a JWT

https://gerrit.wikimedia.org/r/1261545

Mentioned in SAL (#wikimedia-operations) [2026-03-26T20:06:29Z] <kamila@deploy1003> Started scap sync-world: Backport for [[gerrit:1261545|Wrap 'centralauthtoken' in a JWT (T420280)]], [[gerrit:1261470|Enable $wgTempCategoryCollations for testwiki. (T419274 T419049)]]

Mentioned in SAL (#wikimedia-operations) [2026-03-26T20:25:05Z] <kamila@deploy1003> matmarex, kamila: Backport for [[gerrit:1261545|Wrap 'centralauthtoken' in a JWT (T420280)]], [[gerrit:1261470|Enable $wgTempCategoryCollations for testwiki. (T419274 T419049)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-03-26T20:44:01Z] <kamila@deploy1003> Finished scap sync-world: Backport for [[gerrit:1261545|Wrap 'centralauthtoken' in a JWT (T420280)]], [[gerrit:1261470|Enable $wgTempCategoryCollations for testwiki. (T419274 T419049)]] (duration: 37m 32s)

Change #1259242 merged by jenkins-bot:

[operations/deployment-charts@master] rest gateway: add support for centralauthtoken

https://gerrit.wikimedia.org/r/1259242

That patch did not resolve the issue yet, we're working on a follow-up fix.

daniel renamed this task from Authenticated cross-origin requests are being throttled as if unauthenticated to Authenticated cross-origin requests are being throttled as if unauthenticated (centralauth).Apr 1 2026, 12:49 PM

Change #1266237 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] rest gateway: defined authed-user class

https://gerrit.wikimedia.org/r/1266237

Change #1266237 merged by jenkins-bot:

[operations/deployment-charts@master] rest gateway: define authed-user class

https://gerrit.wikimedia.org/r/1266237

The latest patch should have fixed the issue.

I'm re-opening this ticket, because it looks like we missed an important aspect: ext.centralauth.ForeignApi.js will make requests of the form action=query&format=json&origin=https%3A%2F%2Fen.wikipedia.org&meta=userinfo%7Ctokens to see if it needs a centralauth token. By their nature, if the user is not logged in on the target wiki, these requests will be made unauthenticated, and will count against the unauthenticated limit for the IP address. And will start to fail once that limit is reached. This has been reported as a real issue by users.

To fix this, we need to exempt meta=userinfo|tokens from rate limiting - we are already exempting meta=tokens, see T419130.

For context, I ended up hitting a 429 after just using an incognito tab to search for a string while idling on my main account for a fair bit of time.

@daniel That should not be needed for ForeignApi.js – if that request fails for any reason, then it just assumes that it needs to use a centralauthtoken. I can see that problem coming up in other scenarios, though.

@daniel That should not be needed for ForeignApi.js – if that request fails for any reason, then it just assumes that it needs to use a centralauthtoken.

But it counts against the anon rate limit, causing all API requests from the same IP to get blocked.

So, if you are a wiki power user and your are using a gadget that makes heavy use of cross-wiki requests, your requests won't get blocked - but API access for anyone sharing your IP address will be!

@daniel That should not be needed for ForeignApi.js – if that request fails for any reason, then it just assumes that it needs to use a centralauthtoken.

But it counts against the anon rate limit, causing all API requests from the same IP to get blocked.

So, if you are a wiki power user and your are using a gadget that makes heavy use of cross-wiki requests, your requests won't get blocked - but API access for anyone sharing your IP address will be!

I did a bit of digging around, and the action=query&format=json&origin=https%3A%2F%2Fen.wikipedia.org&meta=userinfo%7Ctokens API is being hit by the mark-locked userscript whenever it is initializing the ForeignApi, which subsequently hits the checkForeignLogin code path, which makes this call.

Noting that I've made the following changes to mark-locked now, which should mitigate the problem a fair bit by reducing the number of such calls to one per page load instead of N, where N is the number of users, and assign the request a unique user agent, which should bump it to a higher rate limit tier.

daniel triaged this task as High priority.Apr 8 2026, 8:15 AM

Change #1255731 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] rest gateway: prevent abuse of exempt api modules

https://gerrit.wikimedia.org/r/1255731

Change #1255731 merged by jenkins-bot:

[operations/deployment-charts@master] rest gateway: prevent abuse of exempt api modules

https://gerrit.wikimedia.org/r/1255731

Change #1270514 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] rest gateway: handle percent-escaped pipes in query params

https://gerrit.wikimedia.org/r/1270514

Change #1270514 merged by jenkins-bot:

[operations/deployment-charts@master] rest gateway: handle percent-escaped pipes in query params

https://gerrit.wikimedia.org/r/1270514

Logs show no blocked login requests after the latest patch was deployed today.

Noting that I've made the following changes to mark-locked now, which should mitigate the problem a fair bit by reducing the number of such calls to one per page load instead of N, where N is the number of users, and assign the request a unique user agent, which should bump it to a higher rate limit tier.

Thank you for doing that! It looks like that actually got rid of 90% of the problematic calls already...

image.png (242×1 px, 48 KB)