Create a Convention for Constructive bots:
- if user agent contains (eg. WikiBot) then mark agent_type = bot
Change the agent_type classification to include an exception for the convention above ^ (eg. WikiBot)
- and then additionally classify any user agent with *[Bb][Oo][Tt]* as agent_type = spider (most of these bots are spiders)
- Start filtering bots. Bob's code: https://github.com/cervisiarius/wikimedia/blob/master/navigation_trees/src/main/java/org/wikimedia/west1/traces/GroupAndFilterMapper.java
- Timebox this to avoid exceeding point limit.
- Add a UDF wrapping up manual bot detection for people to use in addtition to agent_type.