Background
The Experiment Platform team have verified that the Experimentation Platform is capable of running a test on all visitors using the JS SDK (see T392313: [Epic] SDS 2.4.11 Run a Synthetic A/A Experiment and most recent update to Hypothesis SDS2.4.11 Synthetic A/A experiment / e2e testing of edge uniques for data collection in Asana). We would like to verify that it is capable of running a synthetic A/A test on logged-in visitors only using the JS SDK.
To do this we will duplicate and tweak the experiment configuration for T395342: Configure synthetic A/A experiment in xLab:
| Detail | Configuration |
|---|---|
| Traffic allocation | English Wikipedia, 1% |
| Identifier type | mw-user |
| Duration | 14 days |
| Experiment name (machine-readable name) | "Logged-in Synthetic A/A Test (JS SDK)" (synth-aa-test-mw-js) |
| Treatment group name (machine-readable name) | "Control 2" (control-2) "No changes for subjects in this group." |
| Instrumentation | PageVisit, producing action: "page-visited" events client-side |
| Metrics | "Page visits", "Page visits per hour" |
Open Questions
Question 1: Do we want to reuse the PageVisit instrument or create a new one so that this task and T397138: Run a second synthetic A/A test aren't mutually exclusive?
Answer 1: Let's rewrite ext.wikimediaEvents/xLab/pageVisit.js to be:
mw.loader.using( 'ext.xLab' ).then( () => { mw.xLab.getExperiment( 'sds2-4-11-synth-aa-test-2' ).send( 'page-visited', { instrument_name: 'PageVisit' } ); mw.xLab.getExperiment( 'synth-aa-test-mw-js' ).send( 'page-visited', { instrument_name: 'PageVisit' } ); } );
AC
- The PageVisit experiment/instrument is updated to reference the synth-aa-test-mw-js machine-readable name
- The experiment's configuration is present in
- Experiment is registered for automated analysis in experiments registry
- Experiment is documented at https://wikitech.wikimedia.org/wiki/Metrics_Platform/List_of_active_and_archived_experiments
