Page MenuHomePhabricator
Create Task
Maniphest T125731

Add regexps that match the bots that follow the User-Agent policy {hawk}
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Currently, the PageviewDefinition.java does not have regular expressions that match the bots that follow the conventions specified in https://meta.wikimedia.org/wiki/User-Agent_policy. We should implement this, and tag them as "bot" (or "spider" :-), it's a low hanging fruit I think.

Details

SubjectRepoBranchLines +/-
Implement the Wikimedia User Agent policy in setting agent typeanalytics/refinery/sourcemaster+77 -189
Customize query in gerrit

Event Timeline

mforns created this task.Feb 3 2016, 11:04 PM
mforns raised the priority of this task from to Needs Triage.
mforns updated the task description. (Show Details)
mforns added a project: Analytics.
mforns subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 3 2016, 11:04 PM
jayvdb subscribed.Feb 3 2016, 11:14 PM
elukey subscribed.Feb 4 2016, 8:51 AM
Milimetric triaged this task as Medium priority.Feb 4 2016, 6:08 PM
Milimetric moved this task from Incoming to Analytics Query Service on the Analytics board.
Milimetric set Security to None.
Milimetric subscribed.

research whether the new expressions that we're adding have a match on the cluster

• Nuria edited projects, added Analytics-Kanban; removed Analytics.Feb 25 2016, 5:34 PM
• Nuria subscribed.

Moving this task to kanban so it is up for grabs given that it is pretty small

Milimetric set the point value for this task to 3.Mar 3 2016, 5:38 PM
• madhuvishy claimed this task.Mar 4 2016, 4:32 PM
• madhuvishy moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot subscribed.Mar 8 2016, 7:33 AM

Change 275760 had a related patch set uploaded (by Madhuvishy):
Implement the Wikimedia User Agent policy in setting agent type

https://gerrit.wikimedia.org/r/275760

gerritbot added a project: Patch-For-Review.Mar 8 2016, 7:33 AM
gerritbot added a comment.Mar 8 2016, 6:56 PM

Change 275760 merged by Joal:
Implement the Wikimedia User Agent policy in setting agent type

https://gerrit.wikimedia.org/r/275760

• madhuvishy moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.Mar 8 2016, 6:58 PM
JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Mar 9 2016, 1:10 PM
• Nuria closed this task as Resolved.Mar 22 2016, 7:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL