Page MenuHomePhabricator

Measure actual effect size from the first A/B test
Closed, ResolvedPublic2 Estimated Story Points

Description

The inclusion of prefix-search-only sessions in the dataset for the first A/B test means the size of the effect is actually larger than the initial analysis suggested. Go through and find out how large!

Event Timeline

Ironholds assigned this task to mpopov.
Ironholds raised the priority of this task from to Needs Triage.
Ironholds updated the task description. (Show Details)
Ironholds added a project: Discovery-ARCHIVED.
Ironholds subscribed.
Ironholds set Security to None.

Processed dataset (ab_test_initial_data.RData) doesn't include information on prefix/full searches. Tried running Erik's Python script on the logs but it breaks (see: T108869). Stalled until that's resolved.

Progress update: running Erik's format conversion py script on the logs.

Performed by Oliver as part of the final analysis (T109760)