Verify that data from A/B test on suggester is coming through correctly (on or after 2015-09-17) redux
Closed, ResolvedPublic1 Estimated Story Points
Actions

Assigned To

Authored By

	• Deskana
	Sep 16 2015, 8:08 PM

Description

Now that the test has been restarted, we should verify that the data is coming in correctly and as expected.

Related Objects
Search...

Status	Assigned	Task
Resolved	mpopov	T111858 Analyze results of A/B test on suggester (on or after 2015-09-22)
Resolved	mpopov	T112813 Verify that data from A/B test on suggester is coming through correctly (on or after 2015-09-17) redux
Resolved	EBernhardson	T112585 Fix CompletionSuggestion data collection and re-start the test.
Resolved	mpopov	T111857 Verify that data from A/B test on suggester is coming through correctly (on or after 2015-09-09)
Resolved	EBernhardson	T111078 Run A/B test on the search suggester to measure zero results rate, starting on 2015-09-08
Resolved	EBernhardson	T111091 Allow extensions to change the method used to get suggestion results
Resolved	EBernhardson	T111137 Override core suggester in AB test between current suggestions and the experimental cirrus-suggest api

Event Timeline

• Deskana created this task.Sep 16 2015, 8:08 PM

• Deskana raised the priority of this task from to High.

• Deskana updated the task description. (Show Details)

• Deskana added projects: Discovery-ARCHIVED, Discovery-Analysis (Current work).

• Deskana subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 16 2015, 8:08 PM

• Deskana added a subtask: T112585: Fix CompletionSuggestion data collection and re-start the test..Sep 16 2015, 8:09 PM

• Deskana closed subtask T112585: Fix CompletionSuggestion data collection and re-start the test. as Resolved.

• Deskana renamed this task from Verify that data from A/B test on suggester is coming through correctly (on or after 2015-09-17) to Verify that data from A/B test on suggester is coming through correctly (on or after 2015-09-17) redux.Sep 16 2015, 8:12 PM

• Deskana set Security to None.

• Deskana added a parent task: T111858: Analyze results of A/B test on suggester (on or after 2015-09-22).

mpopov claimed this task.Sep 17 2015, 1:08 AM

mpopov edited a custom field.

mpopov moved this task from Backlog to In progress on the Discovery-Analysis (Current work) board.Sep 17 2015, 7:15 PM

Browsers seem ok https://github.com/wikimedia-research/CompletionSuggestionTest/blob/master/data_validation/browsers.png

Sampling within both wikis seems fine: https://github.com/wikimedia-research/CompletionSuggestionTest/blob/master/data_validation/wiki.png

I don't like that the group proportions within dewiki are just barely within 2 standard deviations of each other. They should be a lot closer to 50/50 than 55/45.

mpopov moved this task from In progress to Done on the Discovery-Analysis (Current work) board.Sep 18 2015, 11:50 PM

P.S. @Ironholds we need to start using error bars for our proportions as in https://raw.githubusercontent.com/wikimedia-research/CompletionSuggestionTest/master/data_validation/wiki.png

• Deskana closed this task as Resolved.Sep 22 2015, 5:28 PM

• Deskana moved this task from Needs triage to Analysis on the Discovery-ARCHIVED board.

• Deskana moved this task from Done to Resolved on the Discovery-Analysis (Current work) board.Sep 23 2015, 4:47 AM

Verify that data from A/B test on suggester is coming through correctly (on or after 2015-09-17) reduxClosed, ResolvedPublic1 Estimated Story PointsActions

Description

Related ObjectsSearch...

Event Timeline

Verify that data from A/B test on suggester is coming through correctly (on or after 2015-09-17) redux
Closed, ResolvedPublic1 Estimated Story Points
Actions

Related Objects
Search...