Moving to the sprint board as T177302 is done and we've already done a portion of this in T176493.

From @chelsyx:

In T176493#3776047, @chelsyx wrote:

Report for the DBN test: https://analytics.wikimedia.org/datasets/discovery/reports/Experiement_with_different_grouping_of_queries_that_get_fed_into_the_DBN.html

In T177520#3774172, @dcausse wrote:

Moving back to backlog as this task actually covers 2 experiments and thought it was new:

• dbn group sizing : 20 and 35 (https://gerrit.wikimedia.org/r/#/c/387586/): do we have a task to run the report on this data?
• grouping by reording query terms

Should we experiment with smaller group size 5?

Depends on the results of the previous AB test. I think since both 20 and 35 were noticably worse than our arbitrarily chosen default of 10, it's worth testing groups with a smaller size.

Should we experiment with the second suggestion to reorder query terms?

I think this is worthwhile. I do wonder if it will invalidate some of the things we learn about query group sizing though. I suppose it really depends on what hidden variable is changing the user behaviour:

• Is less training data from larger group sizes resulting in less optimization? I don't think this is the case, as the offline ndcg@10 of the model shows a larger increase over baseline with the larger query groupings.
• Are the larger minimum dbn groups throwing out important long tail (or middle tail? there is a huge long tail after 10 sessions per group still..) training data? My intuition is that this is whats happening, but I'm not sure how to validate this.

Assuming the second is true it seems sorting the terms may allow us to get even more of the long tail information into the DBN. On the other hand it may have an unintended effect of putting together queries that aren't as related as we hope. I'm optimistic the second-stage of grouping will negate any poor handling here.

I also think this can at least partly be tested offline:

• Measure the number of first and second stage (or only second stage? first might only be interesting but not particularly useful) groups for both methods of normalizing. Might be interesting to also pull basic count statistics (min/max/std, from spark df.describe()) on sessions per group.
• Could perhaps look at a small sampling of the groups to decide if they look to be any better/worse, or if there are obvious cases where wrong things are being grouped together.

Maybe more? not sure.

As discussed in today's sprint planning, we'll need to run this test again with a smaller sample size.

to clarify, the smaller is in the sizes of the groups. Basically 10 (the default) performed noticably better than both grouping sizes we tested (20, 35). We will run again with smaller sizes (tbd, maybe 5, 8 and 15? 12? i dunno)