Page MenuHomePhabricator
Create Task
Maniphest T412585

Epic: Enforce API rate limits (WE5.1.3c)
Open, Needs TriagePublic
Actions

Description

Turn on rate limiting on the REST gateway, based on the setup created for T398919: Epic: API rate limiting dry run (WE5.1.3b).

The goal is to reduce the number of unidentified requests.

For the initial deployment, there will be three tiers of rate limits:

  1. very high or no limit: known clients (google bot, bing bot, etc), trusted networks (WMF, WMCS, WME), community-approved bots (TBD). See https://wikitech.wikimedia.org/wiki/CDN/Backend_api.
  2. moderate limits: authenticated requests (JWT), compliant bots (UA), and organic browser traffic (heuristic)
  3. restrictive limits: all other traffic

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT399291 Epic: API Rate Limiting Architecture
 OpenNoneT412585 Epic: Enforce API rate limits (WE5.1.3c)
 OpendanielT398919 Epic: API rate limiting dry run (WE5.1.3b)
 DeclinedClement_GoubertT402914 Set up a rest-gateway deployment for rate limiting testing
 ResolveddanielT405544 Add support for rate limiting rest gateway routes to api-gateway helm chart
 ResolveddanielT405636 api-gateway helm chart: rest routes should return retry-after when a rate limit applies.
 ResolveddanielT405578 api-gateway helm chart: use JWT to determine rate limit for rest routes
 ResolvedpmiazgaT406624 Create a mediawiki maintenance script for generating JWT tokens
 ResolvedhnowlanT416205 Deploy Production JWT key on staging cluster
 ResolveddanielT408128 api-gateway chart: make cookie name configurable for testing
 ResolveddanielT406498 Test api rate limiting on production cluster
 ResolvedClement_GoubertT406490 Test api rate limiting on staging cluster
 ResolveddanielT405574 api-gateway helm chart: add ability to test rate limiting for rest-gateway routes
 ResolvedpmiazgaT408839 api-gateway chart: make emission of x-ratelimit headers configurable
 ResolveddanielT408183 api-gateway chart: metrics mapping for rerst-gateway
 ResolvedpmiazgaT409155 For better readability rename `fallback_policy` and `x-wmf-user-class`
 ResolveddanielT407999 api rate limiting: Create a helper service for exposing per-user (per-bucket) metrics
 ResolvedbrennenT409821 Move redioscope repo under mediawiki/services
 ResolveddanielT410559 Publish redioscope image to wmf registry for deployment on the aux cluster
 ResolvedClement_GoubertT413999 Create namespace and support files for redioscope
 ResolveddanielT409044 api-gateway chart: add support for per-route rate limit policy overrides
 ResolveddanielT409543 rest-gateway: define a catch-all rate limit
 DeclinedNoneT410143 Exempt network-local traffic from API rate limiting
 OpenNoneT410198 Determine the source of internal requests going through the API gateway.
 OpenJgiannelosT411769 Migrate wikifeeds backend calls away from rest-gateway
 StalledNoneT411771 Migrate PageViewInfo calls away from rest-gateway
 InvalidNoneT411770 Migrate GrowthExperiments away from rest-gateway
 ResolveddanielT410379 Support setting rate limits based on x-trusted-request headers coming from the edge.
 ResolveddanielT410658 rest gateway: for unauthenticated requests from wmcs, use the User_agent header as the rate limit key
 ResolveddanielT411250 rest gateway: Record x-trusted-request and x-provenance headers in access logs
 OpenSLyngshede-WMFT411503 x-provenance header: identify WMCS
 ResolvedtaaviT411590 Publish machine-readable version of Cloud VPS IP space
 ResolvedtaaviT411610 Publish machine-readable information for Toolforge worker IPs
 OpenNoneT412520 Ensure requests originating from InstantCommons on third party wikis doesn't get rate limited too much
 ResolvedClement_GoubertT413876 Switch ratelimit backend redis to instance 6380
 ResolvedClement_GoubertT414333 Use external_services for redis network policies
 DuplicateClement_GoubertT414349 Log rate limits from rest-gateway in webrequests
 ResolveddanielT413183 rest gateway: add support for shadow policies (selective dry run)
 Resolved• matmarexT415588 Add rate limit class for accounts that are in a local bot group on any wiki
 Resolved• matmarexT416541 Automatic global group membership is updated on unrelated local group changes
 Resolved• matmarexT416542 Log entries for automatic global group membership changes are created on local wikis, not Meta
 OpenNoneT417779 rest gateway: enforce rate limits (stage two)
 OpenNoneT417778 rest gateway: enforce rate limits (stage one)
 ResolveddanielT417780 rest gateway: include x-wmf-* headers in response
 ResolveddanielT417781 haproxy: strip x-wmf-* headers from responses
 ResolvedVgutierrezT417825 CDN: include x-is-browser in all requests to the backend
 ResolvedJAllemandouT417864 haproxy: capture x-wmf-* headers in webrequest data set
 ResolvedDAlangi_WMFT415007 Login with `action=login` and bot password does not create a JWT session cookie
 ResolvedTgrT418999 Remove trailing slash in issuer for bot password JWT cookies
 ResolvedJAllemandouT418455 haproxy: capture `x_trusted_request` in webrequest data set
 DuplicateNoneT418458 MW/API + haproxy: capture `x-wmf-auth-type` in webrequest data set
 ResolveddanielT410273 api rate limiting: Assign ratelimit class based on IP range
 ResolveddanielT413186 rest gateway: support multiple rate limit policies per route
 ResolveddanielT418042 rest gateway: use rlc from sessionJwt cookie even when a bearer token is used
 OpenNoneT418835 rest gateway: ensure sessionJwt matches bearer token
 OpenTgrT417833 Set a JWT cookie for OAuth 1 requests and OAuth 2 owner-only requests
 OpenTgrT418606 Include session type in x-analytics header
 Open• matmarexT418953 Add better error message for HTTP 429 errors to mw.Api#getErrorMessage
 OpenNoneT418957 Add client-side logging for non-MediaWiki action API errors (HTTP 429)
 OpenNoneT418969 Rate limiting gateway should allow cross-origin requests from web browsers to read the HTTP 429 response
 OpenBUG REPORTNoneT419034 Custom OAuth 2 error from Wikimedia infrastructure breaks automatic retry of requests
 OpenNoneT419130 Exempt some routes from rate limiting and JWT validity checks in the API gateway

Event Timeline

daniel created this task.Dec 13 2025, 12:15 PM
daniel added a subtask: T398919: Epic: API rate limiting dry run (WE5.1.3b).
• matmarex subscribed.Dec 14 2025, 10:50 PM

I haven't been able to find this in the related tasks: what does the error response you'll get look like? and is there some way to "force" an error response to test client libraries?

Clement_Goubert added a comment.Dec 15 2025, 5:11 PM
In T412585#11458421, @matmarex wrote:

I haven't been able to find this in the related tasks: what does the error response you'll get look like? and is there some way to "force" an error response to test client libraries?

This will be a standard 429 with a Retry-After header, which *should* be handled correctly by most HTTP client libraries.

We also have the option to give more information via X-RateLimit headers but I don't know if that is desirable or supported by any of the other rate limiting enforcers on our stack (haproxy, varnish).

daniel added a comment.Dec 15 2025, 7:44 PM
In T412585#11458421, @matmarex wrote:

is there some way to "force" an error response to test client libraries?

Hm... That is unfortunately not easily done, since the "shadow mode" flag is in the cinfiguration of the rate limiter, it cannot be controlled per request. What we may end up doing is enabling the rate limit only on one route, for testing.

aaron moved this task from Incoming (Needs Triage) to Radar (other teams work) on the MW-Interfaces-Team board.Dec 17 2025, 5:53 PM
Clement_Goubert created subtask T414349: Log rate limits from rest-gateway in webrequests.Jan 12 2026, 3:06 PM
Scott_French edited projects, added ServiceOps new; removed serviceops-deprecated.Jan 13 2026, 8:07 PM
Scott_French moved this task from Inbox to Radar (Awareness) on the ServiceOps new board.
daniel added a subtask: T413183: rest gateway: add support for shadow policies (selective dry run).Jan 16 2026, 10:27 AM
daniel added a subtask: T413186: rest gateway: support multiple rate limit policies per route.
OWresch-WMF edited projects, added MediaWiki-Platform-Team; removed MediaWiki-Platform-Team (Roadmap).Jan 16 2026, 1:55 PM
Clement_Goubert triaged this task as High priority.Jan 16 2026, 4:14 PM
Clement_Goubert raised the priority of this task from High to Needs Triage.

Sorry for the priority noise, I misclicked while triaging.

• matmarex moved this task from Inbox, needs triage to Radar on the MediaWiki-Platform-Team board.Jan 19 2026, 3:28 PM
• matmarex edited projects, added MediaWiki-Platform-Team (Radar); removed MediaWiki-Platform-Team.
• matmarex added a subtask: T415588: Add rate limit class for accounts that are in a local bot group on any wiki.Feb 2 2026, 9:59 PM
ArielGlenn subscribed.Feb 4 2026, 7:17 PM
daniel created subtask T417778: rest gateway: enforce rate limits (stage one).Wed, Feb 18, 1:32 PM
daniel added a subtask: T417779: rest gateway: enforce rate limits (stage two).Wed, Feb 18, 1:35 PM
daniel removed a subtask: T417778: rest gateway: enforce rate limits (stage one).
daniel changed the status of subtask T413186: rest gateway: support multiple rate limit policies per route from Open to In Progress.Wed, Feb 18, 1:48 PM
daniel closed subtask T413183: rest gateway: add support for shadow policies (selective dry run) as Resolved.
• matmarex closed subtask T415588: Add rate limit class for accounts that are in a local bot group on any wiki as Resolved.Tue, Feb 24, 6:43 PM
daniel closed subtask T413186: rest gateway: support multiple rate limit policies per route as Resolved.Fri, Feb 27, 3:46 PM
Raine mentioned this in T418735: Support for API rate limiting.Mon, Mar 2, 12:59 PM
• matmarex created subtask T418953: Add better error message for HTTP 429 errors to mw.Api#getErrorMessage.Tue, Mar 3, 10:58 PM
• matmarex created subtask T418957: Add client-side logging for non-MediaWiki action API errors (HTTP 429).Tue, Mar 3, 11:13 PM
• matmarex added a comment.Wed, Mar 4, 2:01 AM
In T412585#11461635, @daniel wrote:
In T412585#11458421, @matmarex wrote:

is there some way to "force" an error response to test client libraries?

Hm... That is unfortunately not easily done, since the "shadow mode" flag is in the cinfiguration of the rate limiter, it cannot be controlled per request. What we may end up doing is enabling the rate limit only on one route, for testing.

Some way to test this in production would be really helpful to have, especially for complex scenarios like CORS requests (T418969), which are difficult to exactly replicate locally. Maybe there could be a WikimediaDebug option or something.

Tgr added a comment.Wed, Mar 4, 2:40 PM

Create a global group like rate-limit-testers, make it emit an rlc, make the rate limit for that class 0?

A dedicated WikimediaDebug flag would be nicer for sure, but presumably harder to implement.

Krinkle added a subtask: T419034: Custom OAuth 2 error from Wikimedia infrastructure breaks automatic retry of requests.Tue, Mar 10, 4:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits