So here's the thing about null hypothesis significance testing: when we work with huge sample sizes, we're going to be able to declare the tiniest effects to be statistically significant. For example, in the initial analysis we were able to call an effect size of 0.00115 significant because we had N=5,423,876. In stats literature, a small effect is 0.1, a medium effect is 0.3, and a large effect is 0.5.
Sampling even 1% of the total queries will still give us a very large sample size. I think we need to do a proper sample size calculation. We need to say "this is the smallest effect size we'd consider an improvement, this is the power we want to detect that effect size" and then figure out what our sample size should be. From that, we can then calculate the approximate % that we should sample out of the total.
