Page MenuHomePhabricator
Create Task
Maniphest T125731

Add regexps that match the bots that follow the User-Agent policy {hawk}
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Currently, the PageviewDefinition.java does not have regular expressions that match the bots that follow the conventions specified in https://meta.wikimedia.org/wiki/User-Agent_policy. We should implement this, and tag them as "bot" (or "spider" :-), it's a low hanging fruit I think.

Details

ProjectBranchLines +/-Subject
analytics/refinery/sourcemaster+77 -189Implement the Wikimedia User Agent policy in setting agent type
Customize query in gerrit

Event Timeline

mforns created this task.Feb 3 2016, 11:04 PM
mforns raised the priority of this task from to Needs Triage.
mforns updated the task description. (Show Details)
mforns added a project: Analytics.
mforns added a subscriber: mforns.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 3 2016, 11:04 PM
jayvdb added a subscriber: jayvdb.Feb 3 2016, 11:14 PM
elukey added a subscriber: elukey.Feb 4 2016, 8:51 AM
Milimetric triaged this task as Medium priority.Feb 4 2016, 6:08 PM
Milimetric moved this task from Incoming to Analytics Query Service on the Analytics board.
Milimetric set Security to None.
Milimetric added a subscriber: Milimetric.

research whether the new expressions that we're adding have a match on the cluster

Nuria edited projects, added Analytics-Kanban; removed Analytics.Feb 25 2016, 5:34 PM
Nuria added a subscriber: Nuria.

Moving this task to kanban so it is up for grabs given that it is pretty small

Milimetric set the point value for this task to 3.Mar 3 2016, 5:38 PM
• madhuvishy claimed this task.Mar 4 2016, 4:32 PM
• madhuvishy moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot added a subscriber: gerritbot.Mar 8 2016, 7:33 AM

Change 275760 had a related patch set uploaded (by Madhuvishy):
Implement the Wikimedia User Agent policy in setting agent type

https://gerrit.wikimedia.org/r/275760

gerritbot added a project: Patch-For-Review.Mar 8 2016, 7:33 AM
gerritbot added a comment.Mar 8 2016, 6:56 PM

Change 275760 merged by Joal:
Implement the Wikimedia User Agent policy in setting agent type

https://gerrit.wikimedia.org/r/275760

• madhuvishy moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.Mar 8 2016, 6:58 PM
JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Mar 9 2016, 1:10 PM
Nuria closed this task as Resolved.Mar 22 2016, 7:15 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL