Maniphest T192940

Run analysis on query explorer ab test
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	EBernhardson
	Apr 24 2018, 5:32 PM

Description

After the good results in T189843 we ran the same test on 18 wikis this time. AB test was run for T187148 from 20180329 through 20180419. The data is stored in HDFS as the SearchSatisfaction schema.

We ran 3 standard buckets:
control - The currently deployed ML ranker
classic - The classic non-ML ranker
explorer - The new ranker under test

We also ran 2 interleaved buckets:
control-explorer-i: control in A, explorer in B
classic-explorer-i: classic in A, explorer in B

Related Objects

Mentioned Here: T187148: Evaluate features provided by `query_explorer` functionality of ltr plugin
T189843: Analyse results of query explorer AB test

Event Timeline

EBernhardson created this task.Apr 24 2018, 5:32 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 24 2018, 5:32 PM

• chelsyx added a project: Product-Analytics.Apr 24 2018, 7:39 PM

• chelsyx claimed this task.Apr 24 2018, 8:03 PM

• chelsyx triaged this task as Medium priority.

• chelsyx moved this task from Triage to Backlog on the Product-Analytics board.

• chelsyx moved this task from Backlog to Next Up on the Product-Analytics board.

• chelsyx moved this task from Next Up to Doing on the Product-Analytics board.Apr 25 2018, 6:43 PM

Report: https://analytics.wikimedia.org/datasets/discovery/reports/Evaluate_features_provided_by_query_explorer_on_19_wikis.html
I noticed that the ZRR of enwiki on April 2nd is much higher than other days, but it shouldn't affect CTR since we excluded all zero result searches.
Please let me know if you have any question.

Mark this task as resolved. Feel free to reopen it if you have any question.

Run analysis on query explorer ab testClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Run analysis on query explorer ab test
Closed, ResolvedPublic
Actions