Right now only the sample means are stored (for control and treatment groups). This is OK for proportion-type metric where the variance is just mean * (1 - mean) but for mean-type metrics where the sample variance can only be calculated from underlying data, this does not work.
We need sample variances for planning future experiments and doing power analysis, which depends on baseline and the baseline's variance. Crucially, baseline measurements might not be available outside of experiments (especially if the metric requires edge unique-based units).
Acceptance criteria
- wmf_experiments.experiment_results_v1 has a new sample_variances struct column with control and treatment properties, just like sample_means column
- Analytics job stores the sample variances in this column
- Source code updated
- Conda artifact generated and deployed
- Airflow DAG updated to use new artifact
Notes
ALTER TABLE wmf_experiments.experiment_results_v1 ADD COLUMN sample_variances struct<control: double, treatment: double> AFTER sample_means; UPDATE wmf_experiments.experiment_results_v1 SET sample_variances = STRUCT(NULL, NULL) WHERE sample_variances IS NULL;