Background
Originally ACs for T396059: xLab: update validation for traffic per wiki for Experiments, we decided to spin this feature into its own ticket given the product and analytics impacts of allowing for changes in traffic during an active experiment. We've also cut scope to only include changing (increasing only) sampling rate/traffic allocation -- not including additional wikis.
Why? Traffic increases mean that we will be randomly sampling more of the same selected population. Adding new wikis introduces new populations, and that requires specific statistical and analytical approaches. GrowthBook will allow us to support that in the future, so we don't want to build it today.
Description
Add new validation rules to xLab experiments form:
- Allow for increase of sample rates for targeted wikis in activated experiments – Initial design specs
- Disallow changing wikis for experiments - no adding of wikis
- Prohibit destructive changes to traffic size and target wikis while an experiment is active (i.e. no decrease of sample rates and no removal of wikis)
- For Instruments: allow increasing or decreasing traffic size and adding and removing wikis
Acceptance Criteria
- Above validation rules are implemented
- Unit tests are added to capture validation errors
- Relevant documentation is updated (limitations for active experiments are described in https://wikitech.wikimedia.org/wiki/Experimentation_Lab/Conduct_an_experiment#Launch)