Actually @Nuria this task is only the spark, not the oozie that will make the saprk job run regularly.
I'll modify docs once we have the other one (T175844) done.

I updated the datasources to contain interesting metrics and created a new user able to use and create visualisation, but not mess with the config:

Thanks @BBlack for the detailed explanations :)
As for using the full host header value for wikimedia.org, this is already available as part of the per-domain last-access-uniques cookies I think.

@GoranSMilovanovic : You're very welcome :) Thanks for the informative analysis !

Last update: Data is vetted and exactly the same as Ellery's using my last patch for enwiki on month 2017-08 :)

@GoranSMilovanovic : Thanks for investigating and keeping me posted :) Good to know it was not a bug !
I'm also interested in the reason of why there are so many null banners, if you find it :)

@GoranSMilovanovic : A value is null (in our case banner) when not set. Given how we generate data, it means the url parameter has not been extracted successfully. I can't say if it was because of it not being set or something else.

@GoranSMilovanovic : It wouldn't. I have found this: https://pivot.wikimedia.org/#banner_activity_minutely/line-chart/2/EQUQLgxg9AqgKgYWAGgN7APYAdgC5gQAWAhgJYB2KwApgB5YBO1Azs6RpbutnsEwGZVyxALbVeAfQlhSY4AF9kwYhBkdmeANroVazsApU6jFmw55uOfABtSYag2LWqANycBXcV2DMwxBmC8AEwADACMAOwAtCHRIQCccCEhuMmpIQB0ySEAWkbkACbB4XHRQWFJKWnJWcl5isBgAJ5YXsBwAJIAsiASAEoAggByAOIgCoo6quz6xIVG9EysMxaYVgQkhkrGS2aclrwCQqJtEKJYZADmlA26K9535vgYLg7WxDjbi6b3BzZ2Dicrg8XnQzGoYDgLTaAGU4H0OqMjNZqGJyGANLhNMAAOpdAAivRABTsGAYEhh7hEYnJoUiwAAug1mq1eDCQHAJvJ5EzkNoaN9lk8/nxqIIlMI5PgpDI5A0AEbuCAAawhA2mwuUGv0suoACElarAkoCu5HHpeAAFML4hRKZhksDqi0PbW8B0BBYmIX7NaHMXHKXAGWycQNElMN34YjMCDUQoUS4TBnIMAMTxKJgAR08vgkEAw7nRKGx2dzYHzheLErJIicpAAXtQChIyywKwWi4EU5oU+R3NZrEA==
Is that what you were after ?

@GoranSMilovanovic: Retention in Druid for banners is forever (we keep all history), but there is a glitch: after 2 month we re-index the data and remove the "device" and "region" dimensions to prevent to be able to reconstruct history. We have data since 2016-11-29 onward. Does it make sense?

@GoranSMilovanovic : The temporary location is deleted as part of the oozie workflow (https://github.com/wikimedia/analytics-refinery/blob/master/oozie/banner_activity/druid/daily/workflow.xml#L166).
Please have a look at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Oozie for more on oozie :)

We won't upgrade pivot.

Update on this to gring @Shilad up to speed:

• Original code from Ellery; Mostly in https://github.com/ewulczyn/wiki-clickstream, some sqooping and hive cleaning in https://github.com/ewulczyn/wmf
• Raw data import using Sqoop is productionized (see https://github.com/wikimedia/analytics-refinery/blob/master/bin/sqoop-mediawiki-tables for details)
• Initial spark code has been written (see https://gerrit.wikimedia.org/r/#/c/361459/, does the cleaning and the whole thing at once)
• Data vetting is 90% done but not exactly finished
  • using enwiki for the month of 2017-05, TSV formatted data
    • Joal Spark: /user/joal/wmf/data/archive/clickstream/2017-05/wiki_db=enwiki
    • Ellery Hive: /user/hive/warehouse/joal.db/clickstream_enwiki_201705)
  • 3 rows (over ~27M) have switched category (other in Hive computed dataset have become links in spark computed dataset)
  • Spark dataset doesn't fully aggregate links (many times, 2 rows have the same from/to/type and could be aggregated)

Investigated that tonight:

• No error on grafana nor in the logs
• Data is inserted on master correctly (the exact same query @HaeB pasted above gives up-to-date results when querying on master.

Discussed in standup 2017-08-31: Let's use scap to deploy spark-2.1.1 release folder (with small changes in config for logging and hadoop-conf setting) on stat100[345] and analytics1003 (for prod jobs).

Could be due to hadoop overhead.
To investigate.

It makes a long time this task has been open. Wikistats is not far from ready - We've decided to put effort into implementing those important metrics in wikistats fast.