Page MenuHomePhabricator

Running multiple Search A/B tests in parallel
Closed, DeclinedPublic

Description

This is so I don't forget to look into parallel testing per action item from Discovery retrospective 2017-07-13.

As it is, it would be easy to have multiple tests on by specifying the multiple buckets and adjusting the sampling rates accordingly.

But a more robust solution would be to enable explicit per-test sampling rates. I have some ideas for how to do this, just a matter of getting to it.

Event Timeline

mpopov created this task.Jul 27 2017, 8:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 27 2017, 8:53 PM
debt triaged this task as Medium priority.Jul 27 2017, 10:39 PM
debt moved this task from Needs triage to Up Next on the Discovery-Analysis board.
debt moved this task from needs triage to This Quarter on the Discovery-Search board.
debt closed this task as Declined.Sep 21 2017, 8:38 PM
debt added a subscriber: debt.

This is mostly resolved: we know that we can't run A/B tests on the same wiki, and at this time, we don't have a way to do parallel testing.