In the Milestone 3 of the Relative Trending we need to deploy a new Flink application that will read from the "page_view" stream and a baseline with hourly metrics and will produce a new stream "page_trending".
This application needs to run a sliding window over the page_view stream and count the number of views per key (wiki_id + page_id). The initial setup will be a 1h Window sliding every 5 minutes. This means that each viewed page will produce an event every 5 minutes, telling if the page is trending or not.
This window will be joined with a baseline topic, read as a key-value table in "upsert" mode, meaning that only the latest value for each key will be kept. This baseline table has values needed to compute a z-score inside the Window and decide if a page is trending or not.
The first implementation will use these variables and formulas:
- Window size = 1h
- Window sliding = 5 min
- trending will be true if:
- z-score >= 3 AND view_count >= 50
- Pages without baseline (or baseline with less than 10 view per hour) will be considered as avg = 0 and std.dev = 1. This allows us to use the same formula for all pages.
The view_count >= 50 is added to avoid detecting trending on pages with very low view count, for example, a page that has only 3 views per hours, could produce a z-score >= 3 with only 10 views.
Thresholds and window sizes will be added as configuration parameters, so changing them should be easy in the future.
Task is done if:
- Flink application is running in K8s and producing in eqiad.page_trending.
To Be Defined: I have doubts about the messages being produced. Do we want to produce events at the end of each window, or only when trending changes from true to false and from false to true? I'm assuming we prefer to produce messages for each Window for now, as this is an MVP and that will allow us to analyze the results better.
Note: There are around 180M page_ids across all wikis, only ~8M have more than 3 views per hour, and only ~1,5M have more than 10 views per hours. This means that for those ~178M with almost no views, the formula will mark them as trending if they get 50 views in 1h.
New pages won't have baseline at the beginning, they will fall in the same situation, if a new page gets more than 50 views, it will be always marked as Trending.