Verify data pipeline for BM25 AB test
Closed, ResolvedPublic1 Story Points

Description

Verify that the new test is running well.

debt created this task.Oct 5 2016, 7:13 PM
mpopov claimed this task.Oct 27 2016, 10:34 PM
mpopov set the point value for this task to 1.
mpopov added a subscriber: EBernhardson.EditedOct 27 2016, 11:23 PM

Test seems to be going OK and the session counts seem to be even between groups within wiki-source pairs.

SELECT
  event_source AS source,
  wiki,
  event_subTest AS test_group,
  COUNT(DISTINCT(event_searchSessionId)) AS sessions
FROM TestSearchSatisfaction2_15922352
WHERE
  LEFT(event_subTest, 4) = 'bm25'
GROUP BY wiki, source, test_group
ORDER BY source, wiki, test_group;
sourcewikitest_groupsessions
autocompletejawikibm25:control104
autocompletejawikibm25:inclinks_pv88
autocompletethwikibm25:control5
autocompletethwikibm25:inclinks_pv7
autocompletezhwikibm25:control91
autocompletezhwikibm25:inclinks_pv98
fulltextjawikibm25:control58
fulltextjawikibm25:inclinks_pv47
fulltextthwikibm25:control4
fulltextthwikibm25:inclinks_pv8
fulltextzhwikibm25:control57
fulltextzhwikibm25:inclinks_pv58

The only thing that might slightly worry me is the UA % breakdown by test group (within each wiki & source pair):

SET @currcount = NULL, @currvalue = NULL;
SELECT
  SUBSTRING_INDEX(id, ' - ', 1) AS wiki,
  SUBSTRING_INDEX(id, ' - ', -1) AS source,
  user_agent, control_prop, test_prop, total_sessions
FROM (
  SELECT
    id, user_agent, control_prop, test_prop, total_sessions,
    @currcount := IF(@currvalue = id, @currcount + 1, 1) AS rank,
    @currvalue := id AS whatever
  FROM (
    SELECT
      CONCAT(wiki, ' - ', source) AS id, user_agent,
      SUM(IF(test_group = 'bm25:control', 1, 0))/COUNT(1) AS control_prop,
      SUM(IF(test_group = 'bm25:inclinks_pv', 1, 0))/COUNT(1) AS test_prop,
      COUNT(1) AS total_sessions
    FROM (
      SELECT
        wiki, event_source AS source,
        event_searchSessionId as session_id,
        event_subTest AS test_group,
        MIN(userAgent) AS user_agent
      FROM TestSearchSatisfaction2_15922352
      WHERE
        LEFT(event_subTest, 4) = 'bm25'
      GROUP BY test_group, session_id
    ) AS unique_sessions
    GROUP BY id, user_agent
    ORDER BY id, total_sessions DESC
  ) AS counted_useragents
) AS ranked
WHERE rank <= 5
ORDER BY wiki, source;
wikisourceuser_agentcontrol_proptest_proptotal_sessions
jawikiautocompleteMozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko0.33330.666715
jawikiautocompleteMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.47060.529417
jawikiautocompleteMozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko0.40000.600010
jawikiautocompleteMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.143930.46150.538513
jawikiautocompleteMozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko0.78570.214314
jawikifulltextMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.143930.66670.33333
jawikifulltextMozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko0.25000.75004
jawikifulltextMozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101 Firefox/49.00.75000.25004
thwikifulltextMozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.531.00000.00003
zhwikiautocompleteMozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko0.50000.50008
zhwikiautocompleteMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.40000.600010
zhwikiautocompleteMozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.70000.300010
zhwikiautocompleteMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.46670.533315
zhwikiautocompleteMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.30770.692326
zhwikifulltextMozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101 Firefox/49.00.66670.33333
zhwikifulltextMozilla/5.0 (Macintosh; Intel Mac OS X 10_12) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Safari/602.1.500.66670.33333

What I mean is, 30% controls / 70% test breakdown of 26 zhwiki autocomplete users w/ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36" seems...suspicious. I hope some browsers aren't more or less likely to be in one bucket vs the other.

@EBernhardson What do you think? Too soon to tell if there's an issue and that we'll probably see proportions closer to 50/50 as more searches make it into the test?

I'm not really sure :S. This is all the same code we've used before for splitting people into groups though, so I'm hopefull this is currently due to chance and as we get more sessions the buckets will even out wrt UA.

Moving this to the backlog - let's take another look at this early next week to see if things are still a bit lopsided or if they've evened out.

mpopov added a comment.Nov 7 2016, 6:46 PM

OK, we're good :)

sourcewikitest_groupsessions
autocompletejawikibm25:control16388
autocompletejawikibm25:inclinks_pv16351
autocompletethwikibm25:control3416
autocompletethwikibm25:inclinks_pv3416
autocompletezhwikibm25:control13088
autocompletezhwikibm25:inclinks_pv13031
fulltextjawikibm25:control8357
fulltextjawikibm25:inclinks_pv8371
fulltextthwikibm25:control3037
fulltextthwikibm25:inclinks_pv3082
fulltextzhwikibm25:control6746
fulltextzhwikibm25:inclinks_pv6699
wikisourceuser_agentcontrol_proptest_proptotal_sessions
jawikiautocompleteMozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko0.49800.50202548
jawikiautocompleteMozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko0.50570.49433261
jawikiautocompleteMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.48070.51931629
jawikiautocompleteMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.51980.48021814
jawikiautocompleteMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.50300.49701992
jawikifulltextMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.51200.4880209
jawikifulltextMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.143930.45430.5457328
jawikifulltextMozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko0.49140.5086348
jawikifulltextMozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko0.50460.4954646
jawikifulltextMozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko0.50950.4905944
thwikiautocompleteMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.52500.4750400
thwikiautocompleteMozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.44030.5597402
thwikiautocompleteMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.48270.5173781
thwikiautocompleteMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.49830.5017881
thwikiautocompleteMozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.360.50330.4967306
thwikifulltextMozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.30.52330.476786
thwikifulltextMozilla/5.0 (iPhone; CPU iPhone OS 7_1_1 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D201 Safari/9537.530.56320.436887
thwikifulltextMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.48650.5135111
thwikifulltextMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.53440.4656131
thwikifulltextMozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.530.49680.5032624
zhwikiautocompleteMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.49290.5071986
zhwikiautocompleteMozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.49270.50731021
zhwikiautocompleteMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.50490.49511517
zhwikiautocompleteMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.51580.48422439
zhwikiautocompleteMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.51350.48653626
zhwikifulltextMozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko0.49300.5070142
zhwikifulltextMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.143930.53050.4695164
zhwikifulltextMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.43090.5691181
zhwikifulltextMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.51370.4863292
zhwikifulltextMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.360.50250.4975394
debt added a comment.Nov 8 2016, 8:12 PM

Yay, thanks @mpopov !

debt closed this task as Resolved.Nov 8 2016, 8:13 PM