The purpose of this experiment is to investigate an imbalance we're seeing in Reader Growth's Mobile ToC A/B/C test by replicating their setup to help us isolate the source of the imbalance.
The experiment does not need to be long, 2-3 days will do because we're seeing the imbalance at 48 hours since the experiment went live.
Experiment design
- Traffic allocation (same as https://test-kitchen.wikimedia.org/experiment/mobile-toc-abc)
- Arabic Wikipedia, Chinese Wikipedia, French Wikipedia, Indonesian Wikipedia, Vietnamese Wikipedia: 10 %
- English Wikipedia: 0.1 %
- Variations
- Control (control)
- Treatment A (treatment-a)
- Treatment B (treatment-b)
- Duration
- 3 days
Instrumentation
Client-side instrumentation should be gated and loaded by server-side as in https://gerrit.wikimedia.org/g/mediawiki/extensions/ReaderExperiments/+/4ac24c2c57c860bd280709e24228fb534e4c4a6f/src/Hooks.php#259 with certain events being limited to specific groups via ResourceLoader modules that are then loaded dynamically via server-side assignment checking.
Like the mobile ToC experiment, all instrumentation should only be loaded for Minerva skin as determined via server-side check.
| Event | Sent for |
|---|---|
| action: "page_visit" | All 3 groups |
| action: "module_loaded", action_context: "control" | control only |
| action: "module_loaded", action_context: "treatment-a" | treatment-a only |
| action: "module_loaded", action_context: "treatment-b" | treatment-b only |