Page MenuHomePhabricator

Perform a power analysis to figure out sample size for next A/B test
Closed, ResolvedPublic2 Estimated Story Points

Description

So here's the thing about null hypothesis significance testing: when we work with huge sample sizes, we're going to be able to declare the tiniest effects to be statistically significant. For example, in the initial analysis we were able to call an effect size of 0.00115 significant because we had N=5,423,876. In stats literature, a small effect is 0.1, a medium effect is 0.3, and a large effect is 0.5.

Sampling even 1% of the total queries will still give us a very large sample size. I think we need to do a proper sample size calculation. We need to say "this is the smallest effect size we'd consider an improvement, this is the power we want to detect that effect size" and then figure out what our sample size should be. From that, we can then calculate the approximate % that we should sample out of the total.

Event Timeline

mpopov raised the priority of this task from to High.
mpopov updated the task description. (Show Details)
mpopov added a project: Discovery-ARCHIVED.
mpopov subscribed.
mpopov set Security to None.
mpopov moved this task from Backlog to In progress on the Discovery-Analysis (Current work) board.
mpopov edited a custom field.

Given that we're not _doing_ a next A/B test at the moment, is this ticket currently possible?

mpopov moved this task from Backlog to In progress on the Discovery-Analysis (Current work) board.

Adding sample size calculator to wmf package for transparency and ease of use.

I wasn't sure what odds ratio we're hoping for. (Reminder: odds ratio is the ratio of the odds of treatment group "B" to get some results relative to the odds of control group "A". That is, how much more likely is one group to see results than the other.) So far we've been seeing odds ratios very close to 1, which is to say our test/treatment groups have no been more or less likely than the control groups to see some results.

If we have high hopes for the suggestion API to really make an impact, we can shoot for 1.5 times more likely or 2 times more likely. At 98% power, we'd need 3,169 or 1,241 observations to be able to detect an effect of 1.5 or 2 times, respectively. If we lower our expectations to, say, 1.1 (but keep the 98% power), then we'd need a total of 50,821 observations.

For the numerically inclined:

Odds Ratio80%85%90%95%98%
1.12476128324331474099450821
1.26982798793471156014331
1.334773977465457567136
1.421782491291636064470
1.515441766206725563169
1.611821352158319572426
1.79541091127615791957
1.8798913106913221638
1.968778692011381410
260569280910011241

For the visually inclined:

power_analysis.png (800×600 px, 63 KB)

What that translates to in terms of sampling rate is a different question. Do we want to spread the entire N=50,821 over the 7 days we do the test (not recommended) or do we want to sample N every day and then see how the results differ day-by-day to see whether the effect we're seeing is consistent (recommended).


Code because showing your work is a good practice:

library(wmf) # version 0.1.2

x <- matrix(c(1183934, 1170787, 5563138, 5533010), nrow = 2)
colnames(x) <- c("no results", "some results")
rownames(x) <- c("b", "a")
x <- prop.table(x, margin = 1)
y <- as.data.frame(t(sapply(seq(1.1, 2, 0.1), function(OR) {
  sample_size_odds(odds_ratio = OR, x['a', 'some results'],
                   power = c(0.8, 0.85, 0.9, 0.95, 0.98))
})))
rownames(y) <- seq(1.1, 2, 0.1)

Visualization:

library(ggplot2)
library(ggthemes)
z <- data.frame(odds_ratio = seq(1.1, 2, 0.1),
                power = as.vector(sapply(colnames(y), rep, times = 10)),
                sample_size = as.vector(y))
ggplot(aes(x = odds_ratio, y = sample_size, color = power), data = z) +
  geom_line() + geom_point() + theme_fivethirtyeight() +
  ggtitle("Sample size as a function of odds ratio we'd like to detect")

Now submitted as code to the 'wmf' package and being CRd by me.

(I wonder if the wmf package should live in gerrit? Thoughts?)

Fixing code right now in response to comments.

I suspect installing packages from gerrit would be less convenient.

@mpopov For posterity, before I close this, it would be good to get a one sentence summary of what the sample size for our next test should be. I couldn't see that from a cursory glance the back and forth.

@Deskana: copied from my comment on T111078:

We agreed we want to be able to detect that the experimental group is at least 1.5 times more likely to get results than the control group, with 99% power to detect this effect and 95% confidence (meaning that a p-value less than 0.05 will be called significant). We guessed that 65% of the control group is going to get some results, which means that our sample size should be...

R> wmf::sample_size_odds(odds_ratio = 1.5, p_control = 0.65, power = 0.99, conf_level = 0.95, sample_ratio = 1)
Output: 2133

2133 observations, although that's with a guess of 65% (= 35% zero results rate) prevalence within the control group. The final number will change as we figure out a more accurate, enwiki-specific prevalence.

We have decided to go with a sampling rate of 0.01% and then we are going to sub-sample down to the actual sample size after we collect the data.