Page MenuHomePhabricator

Build backend A/B test infrastructure to measure zero result rate of different search ideas
Closed, ResolvedPublic

Description

We need to be able to set a sampling rate and then bucket users within the sample to perform A/B tests. This should only measure the *second* query. If the users initial search returned results they will not be part of these tests. I would like to measure the effectiveness of different secondary queries we can run to ensure something without results ends up having results. Thus this needs to happen on the backend when performing a search.

We are initially measuring the zero results rate, so this should tie into a fairly straight forward EventLogging schema that records some sort of test identifier, what bucket the search was put into, and the number of results that were returned to the user.

It might also be nice if this could run multiple tests at the same time. Setting up individual tests is outside the scope of this ticket, this is just about getting everything in place and having a schema to log it to.

Event Timeline

EBernhardson raised the priority of this task from to Needs Triage.
EBernhardson updated the task description. (Show Details)
EBernhardson added a project: CirrusSearch.
EBernhardson added a subscriber: EBernhardson.
Restricted Application added a project: Discovery. · View Herald TranscriptJul 24 2015, 8:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
EBernhardson updated the task description. (Show Details)Jul 24 2015, 8:56 PM
EBernhardson set Security to None.

Change 227501 had a related patch set uploaded (by EBernhardson):
Move query rewriting into search backend

https://gerrit.wikimedia.org/r/227501

Change 227578 had a related patch set uploaded (by EBernhardson):
Move query rewriting into search backend

https://gerrit.wikimedia.org/r/227578

Change 227612 had a related patch set uploaded (by EBernhardson):
Bare bones helper for AB testing

https://gerrit.wikimedia.org/r/227612

Change 228404 had a related patch set uploaded (by EBernhardson):
Prevent caching of search requests partitipating in AB test

https://gerrit.wikimedia.org/r/228404

Ironholds moved this task from Needs triage to Search on the Discovery board.Aug 4 2015, 8:17 AM

Change 228404 abandoned by EBernhardson:
Prevent caching of search requests partitipating in AB test

Reason:
taking another route without cookies

https://gerrit.wikimedia.org/r/228404

Change 227612 merged by jenkins-bot:
Bare bones helper for AB testing

https://gerrit.wikimedia.org/r/227612

Change 229949 had a related patch set uploaded (by EBernhardson):
Bare bones helper for AB testing

https://gerrit.wikimedia.org/r/229949

Change 229949 merged by jenkins-bot:
Bare bones helper for AB testing

https://gerrit.wikimedia.org/r/229949

Deskana closed this task as Resolved.Aug 13 2015, 9:56 PM
Deskana added a subscriber: Deskana.

Change 227578 merged by jenkins-bot:
Move query rewriting into search backend

https://gerrit.wikimedia.org/r/227578

Change 227501 merged by jenkins-bot:
Move query rewriting into search backend

https://gerrit.wikimedia.org/r/227501

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 31 2015, 5:08 AM