Page MenuHomePhabricator

Growth: power calculations for experiments
Closed, ResolvedPublic

Description

The Growth team wants to use controlled experiments as much as possible to determine the impact of our work on new editor retention. We will likely be measuring second-week and second-month retention.

This task is to do initial power calculations to understand our limitations with experiments in our pilot wikis, Czech and Korean Wikipedias. Given the volume of new editors that are signing up in those wikis, we want to know how long we'll need to run experiments in order to learn our impact. Given that our annual goal is to increase retention by 10%, I think we would want to be able to detect impact of 5%, but that is something we could discuss.

Event Timeline

nettrom_WMF added a subscriber: mpopov.

We've calculated this for our target wikis individually, and now also for analyzing effects of interventions across multiple wikis with multilevel models (using simulations to estimate the statistical power). For the latter, we also simulated scenarios with additional wikis to understand whether those would allow us to detect smaller effect sizes in a reasonable amount of time.

While I'm here, I'd also like to thank @mpopov for his work on developing the R code used for estimating power in the multilevel model scenario, and for his work in reviewing my modifications to ensure the simulated data reflected the Growth Team's experiment scenarios. Thank you!

Aw, you're very welcome, Morten! :D