A basic form of this was written for my initial evaluation, currently located on stat1002.eqiad.wmnet:/a/ebernhardson/spark_feature_log/code/data_process_dbn.py

For posterity also copying it into a pastie: P5191

In initial tests of this I found that grouping together queries with the same normalized query (determined by applying the standard english stemmer from lucene to queries from english wikipedia) are all considered the same query. That may be a bit too naive, but seems like a reasonable first step. Probably need to browse through some examples to see if anything particularly crazy and different seems to be grouped together. We could do a slightly more expensive comparison, where not only does it need to stem to similar results but the returned result sets should be within X% of each other to be considered the "same" query but with slightly different text.

Additionally, duplicate (query, page_id) data should probably be de-duplicated, and a weight column added to indicate how many copies there were after training the DBN. The DBN itself needs to receive each user session as an independent item, but later feature generation and training will not want to deal with all these duplicates.. The weight may or may not be used in the training step (tbd).

Some partial stats about what our distribution of queries looks like. I'll run this again once we have more data, this is only against ~10 days when we plan to use 80-90 days. We will be able to calculate 60 days worth of data in the next day or two, but due to retention on the web requeste logs after that it will only grow one day at a time until 90 days.

Query used: P5263

Initial concern: Using queries with >= 10 sessions is only ~16% of the sessions. Going all the way to 35 is 6.5%. I expect aggregating over a longer time span will bring these percentages up, but how far? The addition of manually labeled data for the long tail will probably be relatively important, and need to have reasonable weights set on it.

with 40 days:

Queries with >= 10 sessions increased to 29%, >= 35 sessions to 15%. It's still calculating up to 60 days worth.

The backfilling job has finished, so we now have click logs for feb 13 - apr 12.

Over 60 days normalized queries with > 10 sessions make up 33.6% of all sessions. queries with > 35 sessions make up 18.3% of all sessions. This will rise a little more as we approach 90 days, but i wouldn't expect much more than perhaps 40% and 20% respectively.

Change 347566 had a related patch set uploaded (by EBernhardson):
[search/MjoLniR@master] Sample input sessions

https://gerrit.wikimedia.org/r/347566

Change 347566 merged by DCausse:
[search/MjoLniR@master] Sample input sessions

https://gerrit.wikimedia.org/r/347566