Page MenuHomePhabricator
Create Task
Maniphest T228709

Make statistics of throttled/denied WDQS clients
Open, MediumPublic
Actions

Description

We'd like to produce some statistics about which clients are frequently throttled/banned and see if there's anything interesting showing up.
Potential sections:

  • By user agent
  • By IP
  • By query pattern?

The goal would be to figure out what we know about throttling - how much we throttle, how bad it is for the users (i.e. is our throttling affecting 1-2 aggressive bots or hundreds of small users?), etc.

Event Timeline

Smalyshev created this task.Jul 23 2019, 12:33 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptJul 23 2019, 12:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Smalyshev triaged this task as Medium priority.Jul 28 2019, 11:03 PM
Smalyshev added a project: Discovery-Wikidata-Query-Service-Sprint.Aug 1 2019, 5:47 AM
Smalyshev removed a project: Discovery-Wikidata-Query-Service-Sprint.Aug 9 2019, 11:18 PM
Gehel moved this task from Incoming to Scaling on the Wikidata-Query-Service board.Apr 29 2020, 12:55 PM
Gehel added a comment.EditedJun 8 2020, 5:17 PM

We do have a Kibana dashboard that could be used as a basis. We might add a way to filter log for banned / throttled requests. We already have aggregation in place for UA and IP. Implied in the description of this ticket is that we don't just want the dashboards and nice graphs, we also want to do some analysis.

Gehel moved this task from Scaling to Analysis on the Wikidata-Query-Service board.Jun 3 2021, 2:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL