Background
The Experiment Platform team ran their first synthetic A/A test on all visitors in T392313: [Epic] SDS 2.4.11 Run a Synthetic A/A Experiment and its subtasks. Unfortunately, we discovered a bug that lead to the loss of nearly half of the analytics events were sent. We are confident that the distribution of the subject IDs in the events that we did receive is representative of the distribution of all subject IDs, we don't know how many distinct subjects participated in the test.
We would like to run a second synthetic A/A test for 1 week for a sample of 1% of traffic to enwiki to establish this number.
To do this we will duplicate and tweak the experiment configuration for T395342: Configure synthetic A/A experiment in xLab:
| Detail | Configuration |
|---|---|
| Traffic allocation | English Wikipedia, 0.1% |
| Identifier type | edge-unique |
| Duration | 7 days |
| Experiment name (machine-readable name) | SDS 2.4.11 Synthetic A/A Test 2 (sds2-4-11-synth-aa-test-2) |
| Treatment group name (machine-readable name) | "Control 2" (control-2) "No changes for subjects in this group." |
| Instrumentation | PageVisit, producing action: "page-visited" events client-side |
| Metrics | "Page visits", "Page visits per hour" |
NOTE: The PageVisit experiment/instrument will have to be updated.
AC
- The experiment's configuration is present in
- Experiment is registered for automated analysis in experiments registry
- Experiment is documented at https://wikitech.wikimedia.org/wiki/Metrics_Platform/List_of_active_and_archived_experiments