There's some discussion in T163788: Implement clickstream & navigation vectors as a regular job that I merged with this task. In summary:

@Shilad offered to contribute time and code. He has experience with PySpark and Oozie.
@Halfak offered to work with @Shilad to set up an NDA so he could work on our backend.
@Nuria raised concerns about the complexity of computing clickstream stuff as a regular job, but thought that navigation vectors should be easy to compute for a given clickstream dataset.

Update on this to gring @Shilad up to speed:

• Original code from Ellery; Mostly in https://github.com/ewulczyn/wiki-clickstream, some sqooping and hive cleaning in https://github.com/ewulczyn/wmf
• Raw data import using Sqoop is productionized (see https://github.com/wikimedia/analytics-refinery/blob/master/bin/sqoop-mediawiki-tables for details)
• Initial spark code has been written (see https://gerrit.wikimedia.org/r/#/c/361459/, does the cleaning and the whole thing at once)
• Data vetting is 90% done but not exactly finished
  • using enwiki for the month of 2017-05, TSV formatted data
    • Joal Spark: /user/joal/wmf/data/archive/clickstream/2017-05/wiki_db=enwiki
    • Ellery Hive: /user/hive/warehouse/joal.db/clickstream_enwiki_201705)
  • 3 rows (over ~27M) have switched category (other in Hive computed dataset have become links in spark computed dataset)
  • Spark dataset doesn't fully aggregate links (many times, 2 rows have the same from/to/type and could be aggregated)

@JAllemandou, thanks for the pointers! I think there's a little confusion on this, though. I volunteered to productionize Navigation Vectors (see T174796). I'm happy to also work on clickstream once this is done, but I think it will take several months to wrap up Navigation Vectors because of my teaching commitments.

Please let me know if this makes sense!

Last update: Data is vetted and exactly the same as Ellery's using my last patch for enwiki on month 2017-08 :)

@Shilad: My idea was that reviewing this could provide you with interesting knowledge of how/where we store data on the cluster. Let me know if you think it could be valuable.

@JAllemandou let's please document this dataset throughly once available, i will add it to the goals of next quarter for visibility

@Shilad: i do not think navigation vectors depends on click tream dataset being completed, does it? i will remove it as a subtask.

@JAllemandou Let's document dataset on https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream (cadence, availability) announce it to analytics@ and reserach list before we close this task.

Actually @Nuria this task is only the spark, not the oozie that will make the saprk job run regularly.
I'll modify docs once we have the other one (T175844) done.

Ahh, my mistake!