We decided to extract features hourly and label data as "automated" using the last 24 hours of data. This data is calculated for computation only so it should probably be on wmf_raw rather than wmf.
Features to be computed:
sessionId session_start session_end session_length_secs number_of_pageviews pageview_ratio_per_min nocookies user_agent_length
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T138207 [Open question] Improve bot identification at scale | |||
| Resolved | None | T238357 Label high volume bot spikes in pageview data as automated traffic | |||
| Resolved | None | T238358 Deploy high volume bot spike detector to hungarian wikipedia | |||
| Resolved | • Nuria | T238360 Hourly Feature extraction for bot detection from webrequest |
Change 552943 had a related patch set uploaded (by Nuria; owner: Nuria):
[analytics/refinery@master] Create table to hold calculations of session features
Change 552943 merged by Nuria:
[analytics/refinery@master] Table and workflow for features computations per actor per hour
Change 562957 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Fix oozie learning/features/actor/hourly
Change 562957 merged by Joal:
[analytics/refinery@master] Fix oozie learning/features/actor/hourly