mysql:research@analytics-store.eqiad.wmnet [log]> select count(distinct event_searchSessionId), event_subTest from TestSearchSatisfaction2_15357244 where timestamp between 20160315000000 and 20160323120000 group by event_subTest;

+---------------------------------------+----------------+
| count(distinct event_searchSessionId) | event_subTest  |
+---------------------------------------+----------------+
|                                198228 | NULL           |
|                                178843 | phraseBoostEq1 |
+---------------------------------------+----------------+
2 rows in set (40.69 sec)

select count(distinct event_searchSessionId), event_subTest from TestSearchSatisfaction2_15357244 where timestamp between 20160315000000 and 20160323120000 group by event_subTest;Database changed
mysql:research@analytics-store.eqiad.wmnet [log]> select count(distinct event_searchSessionId), event_subTest from TestSearchSatisfaction2_15357244 where timestamp between 20160315000000 and 20160323120000 and event_source = 'fulltext' group by event_subTest;
+---------------------------------------+----------------+
| count(distinct event_searchSessionId) | event_subTest  |
+---------------------------------------+----------------+
|                                 75212 | NULL           |
|                                 65883 | phraseBoostEq1 |
+---------------------------------------+----------------+

16:06 < JustinO> When evaluating results like there, I'd put the numbers in to the spread sheet I sent you. Though more generically, my flight score cards report the metrics which have obtained statistical 
                 significance. They show the raw metric scores of A & B, then show the gain/loss if (and only if) the change is statistically significant.
16:15 < ebernhardson> i'm not entirely sure how to define the "positive samples" part of the spreadsheet for this latest test, we have a largish number of sessions to work with (i think the doc says ~135k 
                      for the week) but no single metric for success/failure of a session
16:19 < ebernhardson> or is positive samples just the set of possibly effected queries?
16:22 < JustinO> positive samples depend on how you define it (and it's symmetrical too). eg: positive = # of clicks in pos 1-3. sample size = number of queries.
16:23 < JustinO> the symmetrical part means you'll get the same answer of significance if you choose the other side. eg: positive = # of clicks in position 4+.
16:24 < JustinO> or, you may rather measure: sample size = number of clicks
16:24 < JustinO> not number of queries.
16:29 < JustinO> control: samples = total number of clicks in control group. positive samples = number of click in position 1-3 in the control group.
16:29 < JustinO> variation: samples = total number of clicks in test group. positive samples = number of click in position 1-3 in the test group.
16:30 < ebernhardson> ok that makes sense, thanks!
16:36 < JustinO> I don't see the total number of clicks in the experiment. but it does list "134,952 independent full-text search session". Let's assume the control & variation group both got ((134,952 / 
                 2) * 0.7) clicks. The 0.7 is a fudge factor to account for people not clicking on anything in the session (abandonment rate). From Fig. 3, ~72.9% of clicks in the control, and ~73.7% of 
                 clicks in the variation group clicked in position 1-3. (note this is a
16:36 < ebernhardson> we have a huge abandonment rate, so the total clicks works out to
16:36 < JustinO> so you sheet would contain:
16:37 < ebernhardson> | count(1) | sum(if(event_position between 1 and 3, 1, 0)) | event_subTest  |
16:37 < ebernhardson> +----------+-----------------------------------------------+----------------+
16:37 < ebernhardson> |    30299 |                                         20716 | NULL           |
16:37 < ebernhardson> |    21551 |                                         15273 | phraseBoostEq1 |
16:37 < JustinO> Control 47233 34432.857
16:37 < JustinO> Variation 47233 34810.721
16:37 < JustinO> awesome
16:38 < ebernhardson> grr, my screenshot thingy is acting up
16:39 < ebernhardson> but it puts positive at 68.37% control, 70.87% test with a very tiny p value
16:39 < JustinO> yep
16:40 < ebernhardson> so interestingly, by that metric the phrase boost is a positive improvement. small but positive
16:40 < JustinO> correct
16:41 < JustinO> specifically in metric of: number of clicks in pos 1-3.

select count(1), sum(if(event_position between 1 and 3, 1, 0)), event_subTest from log.TestSearchSatisfaction2_15357244 where timestamp between 20160315000000 and 20160323120000 and event_action = 'visitPage' group by event_subTest;

mysql:research@analytics-store.eqiad.wmnet [log]> select count(1), sum(if(event_position between 1 and 3, 1, 0)), event_subTest from log.TestSearchSatisfaction2_15357244 where timestamp between 20160315000000 and 20160323120000 and event_action = 'visitPage' and event_source = 'fulltext' group by event_subTest;
+----------+-----------------------------------------------+----------------+
| count(1) | sum(if(event_position between 1 and 3, 1, 0)) | event_subTest  |
+----------+-----------------------------------------------+----------------+
|    23039 |                                         16944 | NULL           |
|    16250 |                                         12566 | phraseBoostEq1 |
+----------+-----------------------------------------------+----------------+

mysql:research@analytics-store.eqiad.wmnet [log]> select event_subTest, count(1), sum(has_click) from (select event_searchSessionId, event_subTest, sum(if(event_action = 'visitPage', 1, 0)) > 0 as has_click from TestSearchSatisfaction2_15357244 where timestamp between 20160315000000 and 20160323120000 and event_action in ('searchResultPage', 'visitPage') group by event_searchSessionId, event_subTest) x group by event_subTest;
+----------------+----------+----------------+
| event_subTest  | count(1) | sum(has_click) |
+----------------+----------+----------------+
| NULL           |   198038 |          20423 |
| phraseBoostEq1 |   178677 |          17200 |
+----------------+----------+----------------+
2 rows in set (1 min 10.92 sec)

mysql:research@analytics-store.eqiad.wmnet [log]> select event_subTest, count(1), sum(has_click) from (select event_searchSessionId, event_subTest, sum(if(event_action = 'visitPage', 1, 0)) > 0 as has_click from TestSearchSatisfaction2_15357244 where timestamp between 20160315000000 and 20160323120000 and event_action in ('searchResultPage', 'visitPage') and event_source = 'fulltext' group by event_searchSessionId, event_subTest) x group by event_subTest;
+----------------+----------+----------------+
| event_subTest  | count(1) | sum(has_click) |
+----------------+----------+----------------+
| NULL           |    75201 |          16311 |
| phraseBoostEq1 |    65879 |          13619 |
+----------------+----------+----------------+