Add a new Hive data set named edit_hourly following this spec:
https://docs.google.com/document/d/1jzrE3xdyEHed4Ek5ORRedOlEeH-i111hdmG3tBTF8QU
This task includes:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | mforns | T211173 "Edit" equivalent of pageviews daily available to use in Turnilo and Superset | |||
| Resolved | mforns | T220092 Set up edit_hourly data set in Hive |
Change 501197 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery@master] Add edit_hourly oozie job
Change 501328 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery@master] Add edit_hourly to list of tables to be purged of old snapshots
Thanks @Neil_P._Quinn_WMF! Forgot to do that.
As you see, we had a slight change of plans in the implementation.
We encountered and issue in Druid, which does not allow to apply transforms to fields that are not listed as dimensions, for hive tables stored in parquet format.
So we decided to create this intermediate table in Hive called edit_hourly (maybe edit_daily, if we find that hourly reveals poor performance).
This way we won't need to use druid transforms (transforms will happen in Hadoop via HiveSQL).
Also, we can take advantage of having the Hive version of the data set for more detailed querying.
Druid developers are fixing this issue in the new version, but it will still take some time until we upgrade to that.
In any case, it won't harm to have that intermediate table in Hive.
Change 501197 merged by Mforns:
[analytics/refinery@master] Add edit_hourly oozie job
Change 501328 merged by Mforns:
[analytics/refinery@master] Add edit_hourly to list of tables to be purged of old snapshots
Added some docs on Wikitech:
https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Edit_hourly
Please @MNeisler and @Neil_P._Quinn_WMF take a look at docs and update as needed: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Edit_hourly