Page MenuHomePhabricator

FY 25/26 WE 5.4.5: Enforce global rate-limits
Open, HighPublic

Description

We want to start adding systematic per-client rate-limiting at the edge. Each different class of users will get different rate-limits:

  • Logged-in users and requests coming from toolsforge will not be rate-limited at the edge. They will still be subject to our anti-flood limits which are incredibly generous (about 50 rps for a single client)
  • Known bots (that either we've classified or have been submitted via the trusted bots program) will get their own dedicated rate-limiting, which will be the one in the Robots Policy, unless other agreements exist with the bot operator
  • Regular traffic that is likely to come from a browser will get a high rate-limit
  • Traffic that is likely not to come from a browser, per T400270, will get rate-limited at or below the robots policy limit; egregious cases / returning / distributed abusers will be blocked individually.

For now, the limits will ostensibly be quite high as we need to fine-tune parameters in browser detection and such.

I think it would also make sense to start constraining these rate-limits to specific urls, or at least exclude things that are extremely lightweight like /w/load.php or /static from the rate-limit counts, given most scrapers will only request actual articles and no css/other bundles.

Pre-conditions to be able to do separate rate limits:

  • Bring the known bots code to production (T400100)
  • Finish and bring to production the browser detection routines (T400270)
  • Bring to 100% production the new verifiable MW sessions (T398815)
  • Add rate-limiting by unique cookie with a fallback on IP, or user-agent + fingerprints to both HIDDENPARMA and the edge CDN code.

Details

Other Assignee
Joe
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+4 -4
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+6 -0
operations/puppetproduction+6 -0
operations/puppetproduction+6 -0
operations/puppetproduction+6 -0
operations/puppetproduction+6 -0
operations/puppetproduction+7 -0
operations/puppetproduction+6 -0
operations/puppetproduction+2 -2
operations/puppetproduction+3 -0
operations/puppetproduction+6 -0
operations/puppetproduction+5 -54
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+8 -1
operations/puppetproduction+2 -2
operations/puppetproduction+7 -0
operations/puppetproduction+2 -2
operations/puppetproduction+7 -0
operations/puppetproduction+2 -2
operations/puppetproduction+7 -0
operations/puppetproduction+2 -2
operations/puppetproduction+7 -0
operations/puppetproduction+2 -2
operations/puppetproduction+6 -0
operations/puppetproduction+8 -0
operations/puppetproduction+0 -14
operations/puppetproduction+0 -6
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+3 -3
operations/puppetproduction+3 -3
operations/puppetproduction+1 -1
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+27 -0
operations/puppetproduction+8 -4
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1212074 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable unid and browser flags rate limits in drmrs

https://gerrit.wikimedia.org/r/1212074

Change #1212073 merged by Fabfur:

[operations/puppet@production] cache::text: enable unid and browser flags rate limits in magru

https://gerrit.wikimedia.org/r/1212073

Change #1212074 merged by Fabfur:

[operations/puppet@production] cache::text: enable unid and browser flags rate limits in drmrs

https://gerrit.wikimedia.org/r/1212074

Change #1212093 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: remove custom ratelimit for cp7001

https://gerrit.wikimedia.org/r/1212093

Change #1212093 merged by Fabfur:

[operations/puppet@production] hiera: remove custom ratelimit for cp7001

https://gerrit.wikimedia.org/r/1212093

Change #1212187 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: revert rate_limiting_flags in drmrs and magru

https://gerrit.wikimedia.org/r/1212187

Change #1212187 merged by Fabfur:

[operations/puppet@production] cache::text: revert rate_limiting_flags in drmrs and magru

https://gerrit.wikimedia.org/r/1212187

Change #1224058 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable rate limit in cache::text

https://gerrit.wikimedia.org/r/1224058

Change #1224058 merged by Fabfur:

[operations/puppet@production] cache::text: enable rate limit in cache::text (ulsfo)

https://gerrit.wikimedia.org/r/1224058

Change #1224104 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable some rate limit in cache::text (magru)

https://gerrit.wikimedia.org/r/1224104

Change #1224105 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable remaining rate limits in cache::text (magru)

https://gerrit.wikimedia.org/r/1224105

Change #1224104 merged by Fabfur:

[operations/puppet@production] cache::text: enable some rate limit in cache::text (magru)

https://gerrit.wikimedia.org/r/1224104

Change #1224105 merged by Fabfur:

[operations/puppet@production] cache::text: enable remaining rate limits in cache::text (magru)

https://gerrit.wikimedia.org/r/1224105

Change #1224609 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text enable auth, known, bot ratelimiting (eqsin)

https://gerrit.wikimedia.org/r/1224609

Change #1224614 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (eqsin)

https://gerrit.wikimedia.org/r/1224614

Change #1224616 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable auth, known, bot ratelimiting (codfw)

https://gerrit.wikimedia.org/r/1224616

Change #1224617 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (codfw)

https://gerrit.wikimedia.org/r/1224617

Change #1224619 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable auth, known, bot ratelimiting (drmrs)

https://gerrit.wikimedia.org/r/1224619

Change #1224620 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (drmrs)

https://gerrit.wikimedia.org/r/1224620

Change #1224621 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable auth, known, bot ratelimiting (eqiad)

https://gerrit.wikimedia.org/r/1224621

Change #1224622 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (eqiad)

https://gerrit.wikimedia.org/r/1224622

Change #1224624 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable auth, known, bot ratelimiting (esams)

https://gerrit.wikimedia.org/r/1224624

Change #1224626 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (esams)

https://gerrit.wikimedia.org/r/1224626

Change #1224609 merged by Fabfur:

[operations/puppet@production] cache::text enable auth, known, bot ratelimiting (eqsin)

https://gerrit.wikimedia.org/r/1224609

Change #1224614 merged by Fabfur:

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (eqsin)

https://gerrit.wikimedia.org/r/1224614

Change #1224616 merged by Fabfur:

[operations/puppet@production] cache::text: enable auth, known, bot ratelimiting (codfw)

https://gerrit.wikimedia.org/r/1224616

Change #1224617 merged by Fabfur:

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (codfw)

https://gerrit.wikimedia.org/r/1224617

Change #1224619 merged by Fabfur:

[operations/puppet@production] cache::text: enable auth, known, bot ratelimiting (drmrs)

https://gerrit.wikimedia.org/r/1224619

Change #1224620 merged by Fabfur:

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (drmrs)

https://gerrit.wikimedia.org/r/1224620

Change #1224621 merged by Fabfur:

[operations/puppet@production] cache::text: enable auth, known, bot ratelimiting (eqiad)

https://gerrit.wikimedia.org/r/1224621

Change #1224622 merged by Fabfur:

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (eqiad)

https://gerrit.wikimedia.org/r/1224622

Change #1224624 merged by Fabfur:

[operations/puppet@production] cache::text: enable auth, known, bot ratelimiting (esams)

https://gerrit.wikimedia.org/r/1224624

Change #1224626 abandoned by Fabfur:

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (esams)

Reason:

delaying cleanup in a later commit

https://gerrit.wikimedia.org/r/1224626

Change #1224725 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (esams)

https://gerrit.wikimedia.org/r/1224725

Change #1224727 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::text: cleanup rate_limiting_flags

https://gerrit.wikimedia.org/r/1224727

Change #1224725 merged by Fabfur:

[operations/puppet@production] cache::text: enable unid, browser ratelimiting (esams)

https://gerrit.wikimedia.org/r/1224725

Change #1224727 merged by Fabfur:

[operations/puppet@production] cache::text: cleanup rate_limiting_flags

https://gerrit.wikimedia.org/r/1224727

Change #1224823 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] cache::text: partial revert of rate_limiting_flags (esams)

https://gerrit.wikimedia.org/r/1224823

Change #1224978 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] cache::text: add eqiad and codfw WMCS public addresses to extra_trust

https://gerrit.wikimedia.org/r/1224978

Change #1224978 merged by Giuseppe Lavagetto:

[operations/puppet@production] cache::text: add eqiad and codfw WMCS public addresses to extra_trust

https://gerrit.wikimedia.org/r/1224978

Change #1225103 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] varnish: remove unit from retry-after header value

https://gerrit.wikimedia.org/r/1225103

Change #1225103 merged by Scott French:

[operations/puppet@production] varnish: remove unit from retry-after header value

https://gerrit.wikimedia.org/r/1225103

Change #1228563 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (magru)

https://gerrit.wikimedia.org/r/1228563

Change #1228568 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (ulsfo)

https://gerrit.wikimedia.org/r/1228568

Change #1228571 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (eqsin)

https://gerrit.wikimedia.org/r/1228571

Change #1228572 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (codfw)

https://gerrit.wikimedia.org/r/1228572

Change #1228573 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (drmrs)

https://gerrit.wikimedia.org/r/1228573

Change #1228574 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (eqiad)

https://gerrit.wikimedia.org/r/1228574

Change #1228575 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (esams)

https://gerrit.wikimedia.org/r/1228575

Change #1228563 merged by Vgutierrez:

[operations/puppet@production] cache::upload: enable global ratelimiting (magru)

https://gerrit.wikimedia.org/r/1228563

Change #1228568 merged by Vgutierrez:

[operations/puppet@production] cache::upload: enable global ratelimiting (ulsfo)

https://gerrit.wikimedia.org/r/1228568

Change #1228571 merged by Vgutierrez:

[operations/puppet@production] cache::upload: enable global ratelimiting (eqsin)

https://gerrit.wikimedia.org/r/1228571

Change #1228572 merged by Vgutierrez:

[operations/puppet@production] cache::upload: enable global ratelimiting (codfw)

https://gerrit.wikimedia.org/r/1228572

Change #1228573 merged by Vgutierrez:

[operations/puppet@production] cache::upload: enable global ratelimiting (drmrs)

https://gerrit.wikimedia.org/r/1228573

Change #1228574 merged by Vgutierrez:

[operations/puppet@production] cache::upload: enable global ratelimiting (eqiad)

https://gerrit.wikimedia.org/r/1228574

Change #1228575 merged by Vgutierrez:

[operations/puppet@production] cache::upload: enable global ratelimiting (esams)

https://gerrit.wikimedia.org/r/1228575

Change #1236679 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting for auth and bot (magru)

https://gerrit.wikimedia.org/r/1236679

Change #1236679 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (magru)

https://gerrit.wikimedia.org/r/1236679

Change #1236701 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (ulsfo)

https://gerrit.wikimedia.org/r/1236701

Change #1236702 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (eqsin)

https://gerrit.wikimedia.org/r/1236702

Change #1236703 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (codfw)

https://gerrit.wikimedia.org/r/1236703

Change #1236704 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (drmrs)

https://gerrit.wikimedia.org/r/1236704

Change #1236705 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (eqiad)

https://gerrit.wikimedia.org/r/1236705

Change #1236706 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (esams)

https://gerrit.wikimedia.org/r/1236706

Change #1236701 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (ulsfo)

https://gerrit.wikimedia.org/r/1236701

Change #1236702 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (eqsin)

https://gerrit.wikimedia.org/r/1236702

Change #1236703 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (codfw)

https://gerrit.wikimedia.org/r/1236703

Change #1236704 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (drmrs)

https://gerrit.wikimedia.org/r/1236704

Change #1236705 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (eqiad)

https://gerrit.wikimedia.org/r/1236705

Change #1236706 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting for bot (esams)

https://gerrit.wikimedia.org/r/1236706

Change #1237241 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (magru)

https://gerrit.wikimedia.org/r/1237241

Change #1237242 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (ulsfo)

https://gerrit.wikimedia.org/r/1237242

Change #1237243 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (eqsin)

https://gerrit.wikimedia.org/r/1237243

Change #1237244 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (codfw)

https://gerrit.wikimedia.org/r/1237244

Change #1237245 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (drmrs)

https://gerrit.wikimedia.org/r/1237245

Change #1237246 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (eqiad)

https://gerrit.wikimedia.org/r/1237246

Change #1237247 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: enable global ratelimiting (esams)

https://gerrit.wikimedia.org/r/1237247

Change #1237241 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting (magru)

https://gerrit.wikimedia.org/r/1237241

Change #1239052 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] varnish::upload: raise limit for unid browsers

https://gerrit.wikimedia.org/r/1239052

Change #1239052 merged by Fabfur:

[operations/puppet@production] varnish::upload: raise limit for unid browsers

https://gerrit.wikimedia.org/r/1239052

Change #1239141 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] varnish::upload: raise limit for unid browsers (hits)

https://gerrit.wikimedia.org/r/1239141

Change #1239141 merged by Fabfur:

[operations/puppet@production] varnish::upload: raise limit for unid browsers (hits)

https://gerrit.wikimedia.org/r/1239141

Change #1239703 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: increase global request limit on upload (browser)

https://gerrit.wikimedia.org/r/1239703

Change #1239703 merged by Fabfur:

[operations/puppet@production] cache::upload: increase global request limit on upload (browser)

https://gerrit.wikimedia.org/r/1239703

Change #1237242 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting (ulsfo)

https://gerrit.wikimedia.org/r/1237242

Change #1237243 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting (eqsin)

https://gerrit.wikimedia.org/r/1237243

Change #1237244 merged by Fabfur:

[operations/puppet@production] cache::upload: enable global ratelimiting (codfw)

https://gerrit.wikimedia.org/r/1237244