Create an Oozie coordinator that executes the session length query on top of event.mediawiki_client_session_tick table on a daily basis,
and generates the "sessionized" intermediate table that will be queriable by Presto to generate session length dashboards.
Description
Description
Details
Details
Related Changes in Gerrit:
Related Objects
Related Objects
Event Timeline
Comment Actions
Change 664885 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery@master] [WIP] Add oozie job for session length computation
Comment Actions
I finished and tested the Oozie job, and seems to be working fine!
@Mayakp.wiki and I sync'ed up on data checks and, while Maya is making sure the data looks good, I will push the code for CR with my team.
One question to @jlinehan and @Mholloway: When should the Oozie job start computing? Do we want to wait until we have the events flowing from all wikis at 1/10, or should we start already? Thanks!
Comment Actions
I ran QA checks similar to T271455 but for data from Jan 6 to Jan 21 of the raw table (since mforns.session_length_daily (intermediate table) has data loaded from mid of Jan 5th)
The data in the intermediate table mforns.session_length_daily looks good and findings are similar to what we found earlier. Available in this Notebook.
Comment Actions
Change 664885 merged by Mforns:
[analytics/refinery@master] Add oozie job for session length computation
Comment Actions
Change 670239 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery@master] Preemptively add weight field to session length intermediate table
Comment Actions
Change 670239 abandoned by Mforns:
[analytics/refinery@master] Preemptively add weight field to session length intermediate table
Reason:
Doing this later, when we have the sampling rate field in the schema.