Page MenuHomePhabricator

api-gateway helm chart: add ability to test rate limiting for rest-gateway routes
Closed, ResolvedPublic

Description

This is the initial stepping stone for T405544. It enables manual testing and verification of the ratelimiting setup in a production environment, without imposing limits. For this purpose:

  • rate limiting should be disabled per default
  • rest routes should be able to opt into rate limiting
  • the user identity will be taken from the centralauth-user cookie
  • rate limiting should be applied only if a spcial header is sete by the client, providing the cuser "class" that determins the rate limits
  • rate limiting is implemented using the same Redis backend that we also use to implement rate limiting for api.wikimedia.org

Solution

Ratelimiting is based on the existence of two headers:

  • x-wmf-user-id -> a user idenfier
  • x-wmf-user-class -> user type, anon or cookie-user at this moment.

The x-wmf-user-id header gets injected by Lua script. Second header x-wmf-user-class will be also injected by Lua script, but currently this logic is commented out and it is expected for the caller to pass this header to remotely enable the rate limitting.

The solution was tested locally, by sending curl -s -H "x-wmf-user-class: anon" localhost:8087/enwiki/v1/page/mobile-html/a/b request gets ratelimited. When sent without the x-wmf-user-class header, the ratelimiting logic is skipped.

At this moment we can easily check the envoy response headers. When the request is goes trough ratelimiting, envoy injects couple headers to the response:

By default we get:

x-ratelimit-limit: 2, 2;w=60
x-ratelimit-remaining: 1
x-ratelimit-reset: 40

And when request is blocked, the response becomes http 429 with

x-envoy-ratelimited: true
x-ratelimit-limit: 2, 2;w=60
x-ratelimit-remaining: 0
x-ratelimit-reset: 12

Event Timeline

Change #1189447 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] Add rate limiting for REST gateway (WIP)

https://gerrit.wikimedia.org/r/1189447

Change #1191318 had a related patch set uploaded (by Daniel Kinzler; author: Polishdeveloper):

[operations/deployment-charts@master] apigw chart: for rest call ratelimit only when x-wmf-user-class header is present

https://gerrit.wikimedia.org/r/1191318

pmiazga updated the task description. (Show Details)

Note: The response headers about the ratelimit configuration (remaining/reset) comes from a envoy config flag:

enable_x_ratelimit_headers: DRAFT_VERSION_03

Note: The response headers about the ratelimit configuration (remaining/reset) comes from a envoy config flag:

enable_x_ratelimit_headers: DRAFT_VERSION_03

I filed T405636: api-gateway helm chart: rest routes should return retry-after when a rate limit applies.

Change #1189447 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: Add rate limiting for REST gateway

https://gerrit.wikimedia.org/r/1189447

Change #1199331 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] api-gateway: Release patch for ratelimit test

https://gerrit.wikimedia.org/r/1199331

Change #1191318 abandoned by Clément Goubert:

[operations/deployment-charts@master] api-gateway: rest gw should call ratelimit only when x-wmf-user-class header is present

Reason:

Stacked in Iec9ddec04c7d1a82bf48d2a0e54bbec2aa29f4dc

https://gerrit.wikimedia.org/r/1191318