Adminstrator sets API rate limit for clients without an API key
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• eprodromou
	Feb 26 2020, 7:38 PM

Description

"As an Administrator, I want to limit the number of API calls that can be made without an OAuth 2.0 API key during a particular period of time, so I can plan for and manage API traffic usage."

This ticket was originally to have a global pool for all API calls without an OAuth 2.0 client ID. However, based on a discussion on the wiki page, we decided to have a pool for each IPv4 or IPv6 calling address.

This doesn't keep the total pool of calls bounded at a reasonable size, which is a bummer for capacity estimation, but it's somewhat more useful for client developers to estimate their effort.

So, this task is to have a rate limit per IP address only for requests that are made without an OAuth 2.0 ID. This can be very simple; no need to do things like identify VPNs or IP blocks or TOR nodes or whatever. Just a pool per IP address.

For MVP, a good value is 500 requests/hour per IP address.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
api-gateway: use x-client-ip instead of x-forwarded-for IP	operations/deployment-charts	master	+7 -3
Switch ratelimit service to V3 protocol	operations/deployment-charts	master	+2 -2
Configure ratelimiter to support authenticated/anon limits for api	operations/deployment-charts	master	+90 -21

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• eprodromou	T235270 Wikimedia API Gateway
Resolved	• eprodromou	T255032 Wikimedia API Gateway Public Launch
Resolved	• Pchelolo	T246276 Adminstrator sets API rate limit for clients without an API key

Event Timeline

• eprodromou created this task.Feb 26 2020, 7:38 PM

• eprodromou edited parent tasks, added: T255032: Wikimedia API Gateway Public Launch; removed: T246269: API Rate Limiting.Jun 10 2020, 3:57 PM

I've made this a public-launch task. Supporting developers without a client ID is important for public use, but it's not a new paradigm and not one we need to work on for a beta release.

• eprodromou updated the task description. (Show Details)Jul 27 2020, 11:59 AM

• eprodromou edited projects, added Platform Team Workboards (Green); removed Platform Team Workboards (User Stories).

• eprodromou moved this task from Backlog to Ready on the Platform Team Workboards (Green) board.Jul 27 2020, 12:14 PM

• eprodromou reassigned this task from • eprodromou to hnowlan.Jul 27 2020, 12:59 PM

• eprodromou triaged this task as Medium priority.

• eprodromou renamed this task from Adminstrator sets API limit for clients without an API key to Adminstrator sets API rate limit for clients without an API key.Jul 29 2020, 1:38 PM

• eprodromou updated the task description. (Show Details)

• WDoranWMF added a project: Platform Team Sprints Board (Sprint 0).Jul 31 2020, 12:15 PM

Naike edited projects, added Platform Team Sprints Board (Sprint 1); removed Platform Team Sprints Board (Sprint 0).Aug 7 2020, 10:57 AM

Change 619804 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/deployment-charts@master] Configure ratelimiter to support authenticated/anon limits for api

https://gerrit.wikimedia.org/r/619804

gerritbot added a project: Patch-For-Review.Aug 17 2020, 5:01 PM

• Pchelolo claimed this task.Aug 17 2020, 5:02 PM

• Pchelolo moved this task from Ready to Waiting for Review on the Platform Team Workboards (Green) board.

• Pchelolo added a subscriber: hnowlan.

Change 619804 merged by jenkins-bot:
[operations/deployment-charts@master] Configure ratelimiter to support authenticated/anon limits for api

https://gerrit.wikimedia.org/r/619804

Change 620766 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/deployment-charts@master] Switch ratelimit service to V3 protocol

https://gerrit.wikimedia.org/r/620766

Change 620766 merged by jenkins-bot:
[operations/deployment-charts@master] Switch ratelimit service to V3 protocol

https://gerrit.wikimedia.org/r/620766

Maintenance_bot removed a project: Patch-For-Review.Aug 17 2020, 7:11 PM

The above patches were deployed, however we can't mark this as done:

anon rate limits are not working yet. They depend on a feature @Clarakosi introduced into envoy, but that will only be available in 1.16 that we are patiently waiting for. Once we update, one line needs to be uncommented and it should start working.
authenticated rate limits depend on OAuthRateLimiter.

Moving to blocked.

Naike edited projects, added Platform Team Sprints Board (Sprint 2); removed Platform Team Sprints Board (Sprint 1).Aug 21 2020, 1:13 PM

Change 626146 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] api-gateway: use x-client-ip instead of x-forwarded-for IP

https://gerrit.wikimedia.org/r/626146

gerritbot added a project: Patch-For-Review.Sep 9 2020, 12:34 PM

Change 626146 merged by jenkins-bot:
[operations/deployment-charts@master] api-gateway: use x-client-ip instead of x-forwarded-for IP

https://gerrit.wikimedia.org/r/626146