Page MenuHomePhabricator

Perform analysis of results of A/B test with phrase rescore boost of 1
Closed, ResolvedPublic8 Story Points

Description

After the test has been turned off and a week's data has been collected (T129607), perform a full analysis of the data from the A/B test to determine if the change had its intended effect as defined in T129601.

Event Timeline

Deskana created this task.Mar 11 2016, 1:08 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2016, 1:08 AM
mpopov claimed this task.Mar 24 2016, 11:05 PM
mpopov set the point value for this task to 4.
mpopov moved this task from Backlog to In progress on the Discovery-Analysis (Current work) board.
Deskana triaged this task as Normal priority.Mar 28 2016, 10:26 PM
Deskana moved this task from Needs triage to On Sprint Board on the Discovery board.
mpopov changed the point value for this task from 4 to 8.Mar 31 2016, 5:36 PM

Draft (finally):

I should look into automating this, since other tests will use the same schema and involve the same metrics.

I volunteered to review this. It looks good to me.

@mpopov Can you upload to Commons? Then we can call this done.

mpopov added a comment.Apr 7 2016, 5:08 PM

David, Erik, Trey, and I met this morning to discuss what may have gone wrong. We've decided to re-analyze after filtering out queries that would not have been affected (one word queries) because we suspect those may have skewed the results and made the two groups look the same.