Page MenuHomePhabricator

Rate limit requests in violation of User-Agent policy more aggressively
Open, MediumPublic

Description

Wikimedia's User-Agent policy specifically forbids using generic values for the User-Agent request header.

Apply stricter rate limiting to requests violating the policy.

Details

Related Gerrit Patches:

Event Timeline

ema created this task.Jun 3 2019, 3:04 PM
Restricted Application added a project: Operations. · View Herald TranscriptJun 3 2019, 3:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as Medium priority.Jun 3 2019, 3:04 PM
Paladox added a subscriber: Paladox.Jun 3 2019, 3:06 PM

Change 514017 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache_upload: return HTTP 403 to requests violating UA policy

https://gerrit.wikimedia.org/r/514017

ema moved this task from Triage to Caching on the Traffic board.Jun 3 2019, 3:09 PM

Change 514017 merged by Ema:
[operations/puppet@production] cache_upload: return HTTP 403 to requests violating UA policy

https://gerrit.wikimedia.org/r/514017

For Tech News: Bots and other scripts that do not set an identifiable User-Agent may find their requests blocked until they identify themselves properly.

TheDJ added a subscriber: TheDJ.Jun 4 2019, 3:24 PM

Not sure if it applies here, but please remember that we allow Api-User-Agent as an alternative to User-Agent for Javascript solutions.

ema renamed this task from Return HTTP 403 to requests in violation of User-Agent policy to Rate limit requests in violation of User-Agent policy more aggressively.Jun 5 2019, 2:48 PM
ema updated the task description. (Show Details)

We (Traffic) have decided to continue allowing requests violating the UA policy. Instead of blocking them, we will apply stricter rate limiting to those.

Change 513596 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: cache_upload rate limit

https://gerrit.wikimedia.org/r/513596

Change 513596 merged by Ema:
[operations/puppet@production] varnish: cache_upload miss/pass rate limit

https://gerrit.wikimedia.org/r/513596

TechNews: I've added it to the upcoming edition with this edit, that will be frozen for translation in about 18 hours. Please amend it before then if needed. (And thank you @Legoktm for writing the initial version!). Cheers!