This is an epic to address a number of issues and proposed improvements related to bot detection.
Description
Description
Related Objects
Related Objects
- Mentioned In
- T295073: <Org-Wide Impact> Google Chrome User-Agent Deprecation Impact
T292449: Revisit approach to automated bot detection - Mentioned Here
- T295073: <Org-Wide Impact> Google Chrome User-Agent Deprecation Impact
T326337: Add redirects in bot detection features
T326339: Use uap-core browser-family for bot detection
T292449: Revisit approach to automated bot detection
Event Timeline
Comment Actions
This also relates closely to T292449: Revisit approach to automated bot detection
We discussed this in the Data Engineering Office hours with @Mayakp.wiki
Comment Actions
I recently discovered the public_cloud key in X-Analytics. If I understand it correctly, it might be a simple additional heuristic to just label all of that traffic as likely automated (given that it's coming via Google/Amazon/Microsoft/etc. cloud)? Or would that capture a bunch of proxies too used by actual readers?
I also found T326337 and T326339 which presumably should be linked here too as subtasks?
Comment Actions
We need to improve bot detection as most of the traffic seems to be coming from a few UAs (like Chrome 111 on Android 10) look pretty clearly like human traffic, which also means we need to re-prioritize our work on T295073 as the cause seems to be that the implementation by Chrome of the more generic user-agent seems to finally be rolling out (timeline) and so is breaking the bot detection pipelines in pretty significant ways .
In 2023 pageview data, we are seeing spikes in automated traffic that are now affecting external (search engine) referrer traffic (chart)
We need to improve bot detection as most of the traffic seems to be coming from a few UAs (like Chrome 111 on Android 10) look pretty clearly like human traffic, which also means we need to re-prioritize our work on T295073 as the cause seems to be that the implementation by Chrome of the more generic user-agent seems to finally be rolling out (timeline) and so is breaking the bot detection pipelines in pretty significant ways .
Comment Actions
Update:
Automated traffic spikes we saw in March continued into April 2023. We observed that automated traffic which usually shows up as None or Direct referer traffic is dropping and subsequently increasing in External Search Engine referer traffic. (chart)
