Run tests varying the options to the suggestions api
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	EBernhardson
	Jul 28 2015, 4:28 PM

Description

Run the tests and see what our best options are. Decisions on what tests to run are in T107163

Related Objects
Search...

Status	Assigned	Task
Resolved	• Deskana	T107164 Run tests varying the options to the suggestions api
Resolved	• Deskana	T107163 Decide on sets of suggestion api variations to try in production user testing
Resolved	EBernhardson	T106691 Write a script that send queries to the production cluster and collect the results
Declined	Ironholds	T106889 Expose the result of zero result A/B tests on searchdata.wmflabs.org
Resolved	EBernhardson	T106888 Build backend A/B test infrastructure to measure zero result rate of different search ideas
Resolved	EBernhardson	T107171 Create connector porting the A/B test Cirrus logfiles to stat1002 from Fluorine

Event Timeline

EBernhardson created this task.Jul 28 2015, 4:28 PM

EBernhardson raised the priority of this task from to Needs Triage.

EBernhardson updated the task description. (Show Details)

EBernhardson subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 28 2015, 4:28 PM

EBernhardson added subtasks: T107163: Decide on sets of suggestion api variations to try in production user testing, T106889: Expose the result of zero result A/B tests on searchdata.wmflabs.org, T106888: Build backend A/B test infrastructure to measure zero result rate of different search ideas.Jul 28 2015, 4:28 PM

EBernhardson added a project: CirrusSearch.

EBernhardson set Security to None.

Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptJul 28 2015, 4:29 PM

Ironholds added a subtask: T107171: Create connector porting the A/B test Cirrus logfiles to stat1002 from Fluorine.Jul 28 2015, 5:38 PM

Ironholds moved this task from Needs triage to Search on the Discovery-ARCHIVED board.Aug 4 2015, 8:16 AM

Ironholds closed subtask T106889: Expose the result of zero result A/B tests on searchdata.wmflabs.org as Declined.Aug 4 2015, 8:19 PM

EBernhardson updated the task description. (Show Details)Aug 5 2015, 8:48 PM

EBernhardson added a project: Discovery-Search (Current work).

EBernhardson updated the task description. (Show Details)

David, Trey and I talked about this over an email thread for the last few days. We then had a meeting this morning to discuss where we are going with this test and future tests.

The summary is: suggester is already setup quite loosely with respect to how it generates suggestions. The remaining best option to tune is to reduce the confidence of the suggestions which changes the threshold at which a generated suggestion is discarded. David has instrumented the code base to make many suggestion variables, including these two, adjustable on demand for our tests.

EBernhardson moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Aug 5 2015, 11:11 PM

• Deskana closed subtask T106888: Build backend A/B test infrastructure to measure zero result rate of different search ideas as Resolved.Aug 13 2015, 9:56 PM

• Deskana closed subtask T107163: Decide on sets of suggestion api variations to try in production user testing as Resolved.

• ksmith moved this task from Search to On Sprint Board on the Discovery-ARCHIVED board.Aug 27 2015, 8:30 PM

• ksmith added a project: OKR-Work.Sep 1 2015, 6:10 PM

• Deskana closed subtask T107171: Create connector porting the A/B test Cirrus logfiles to stat1002 from Fluorine as Resolved.Sep 9 2015, 2:39 AM

• Deskana closed this task as Resolved.Sep 12 2015, 2:43 AM

• Deskana claimed this task.

• Deskana moved this task from Needs Reporting to Resolved on the Discovery-Search (Current work) board.

• Deskana moved this task from Inbox to Resolved/Invalid/Declined/Legacy on the CirrusSearch board.Dec 31 2015, 5:08 AM

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 31 2015, 5:08 AM