Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables
Open, Stalled, NormalPublic

Description

In T116065: Design aggregate tables to drive Action API reports I designed a series of dimensional rollup tables to make reporting on various interesting Action API metrics easier. I have been running an ETL process using cron, bash, Python, and hive on stat1002 since April 2016 that populates tables for that schema in the bd808 database. This workflow needs to be converted to use standard Analytics tools and move to the wmf database.

Note: now running on stat1005 due to stat1002 decomm.

bd808 created this task.Jun 8 2016, 3:56 PM
Restricted Application added a project: User-bd808. · View Herald TranscriptJun 8 2016, 3:56 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript
bd808 added a comment.Jun 8 2016, 5:41 PM

The scripts I have been using are available at https://github.com/bd808/action-api-analytics

Moving to radar for now, but when you prioritize and define this, we can help code the Oozie jobs that would get this done. Just let us konw.

Milimetric moved this task from Incoming to Radar on the Analytics board.Jul 7 2016, 5:38 PM
bd808 added a comment.Jul 7 2016, 5:42 PM

Moving to radar for now, but when you prioritize and define this, we can help code the Oozie jobs that would get this done. Just let us konw.

I can make time to work on it whenever someone has time to help me. I can probably knock out most of it with just a few pointers to wiki pages and/or gerrit commits that show doing something similar. The tasks that get run are really just /usr/bin/hive with proper year, month, day, hour values supplied as cli args.

oh, sweet, then it should be pretty straightforward. Grab me in IRC or a hangout whenever we're both free. As far as examples, all of our oozie code is here (but I will help you navigate it):

https://github.com/wikimedia/analytics-refinery/tree/master/oozie

And all of our oozie documentation is here (warning: XML is present in concentrations that may burn your eyes):

https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie

bd808 triaged this task as Normal priority.Jul 22 2016, 5:05 PM
bd808 added a comment.Sep 30 2016, 1:15 AM

@Tgr do you have the time and energy to take this task over and finally finish our team goal from Q3 2015/16?

bd808 reassigned this task from bd808 to Tgr.Sep 30 2016, 10:16 PM

I think I have successfully conned convinced @Tgr to take this on when he gets some time.

Same offer to help with oozie applies to @Tgr, of course, and bonus: I'm now a lot better at setting up oozie jobs.

Change 331100 had a related patch set uploaded (by Gergő Tisza):
[WIP] Add Oozie jobs for wmf_raw.ApiAction -> wmf.action_*

https://gerrit.wikimedia.org/r/331100

@Tgr I'm aiming to review this by the end of the week, please ping me if I slip up.

Addshore changed the task status from Open to Stalled.Aug 28 2018, 7:34 AM
Addshore added a subscriber: Addshore.

Just giving this a poke 1 year on as it blocked T174474: ApiAction log in data lake doesn't record Wikibase API actions what's the status here?

Marking as stalled until this is updated.

Addshore moved this task from Backlog to Blocked / Waiting on the User-Addshore board.

ping @Anomie or @Tgr: were either of you interested in finishing this change https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/331100/ or should I consider taking it over when I have some time?

Feel free as far as I'm concerned. I don't know anything about Oozie.

Nuria added a subscriber: Nuria.Aug 31 2018, 5:17 PM

To be fair I do not think we will have any time to work on this soon and I am really of the opinion that it would be best if the api team owns the instrumentation code that instruments the api, that way the code will be properly maintained and changes on api code will be propagated to changes in instrumentation. We are happy to help api developers to push the changes through as needed be.

Tgr added a comment.Sep 21 2018, 1:33 AM

ping @Anomie or @Tgr: were either of you interested in finishing this change https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/331100/

Not anytime soon; I'm unfamiliar enough with Hive and Oozie that even small changes take very long, and don't have the time for it right now.

Milimetric moved this task from Radar to Incoming on the Analytics board.Oct 1 2018, 2:47 PM
Milimetric raised the priority of this task from Normal to Unbreak Now!.
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptOct 1 2018, 2:47 PM
Milimetric lowered the priority of this task from Unbreak Now! to Normal.Oct 1 2018, 2:48 PM
Ottomata moved this task from Incoming to Radar on the Analytics board.Oct 4 2018, 5:16 PM
Ottomata raised the priority of this task from Normal to Needs Triage.
Milimetric triaged this task as Normal priority.Oct 4 2018, 5:18 PM