Background
Based on T287709: [SPIKE] Explore instrumentation for sticky header work and subsequent discussion, we will be creating a schema that tracks whether the user is in the control or test bucket of the sticky header A/B test.
Acceptance criteria
- Setup the A/B test schema.
- Add a configuration option to Vector for controlling whether the sticky header is shown.
- When the Vector experiment is enabled, make use of mediawiki.experiments to determine the bucket (mw.experiments.getBucket) and notify WikimediaEvents that an experiment is running (consider calling mw.track('wikimediaEvents.experiments')).
- The WikimediaEvents extension when notified that an experiment is running should log an event which contains:
- the user session ID
- the experiment group the user is assigned to (A or B). This should come from the value of mw.experiments.getBucket
- the experiment name (which is communicated to it by Vector)
- the user ID.
- If no experiment is running the WikimediaEvents should log nothing to the new schema
Developer notes
The mediawiki experiments module in core, provides a lot of the infrastructure needed here.
mw.experiments.getBucket( {
name: 'My first experiment',
enabled: true,
buckets: {
control: 0.5
A: 0.25,
B: 0.25
}
} )Notes on schema came from the events defined in the sticky header instrumentation spec
QA steps
- For testing event logging locally, the following config will be needed in local settings:
$wgVectorWebABTestEnrollment = [ 'name' => 'vector.sticky_header_2021_11', 'enabled' => true, 'buckets' => [ 'unsampled' => [ 'samplingRate' => 0.1, ], 'control' => [ 'samplingRate' => 0.3, ], 'stickyHeaderDisabled' => [ 'samplingRate' => 0.3, ], 'stickyHeaderEnabled' => [ 'samplingRate' => 0.3, ] ] ];
- Create some new test users to make sure you have one in each group: control, stickyHeaderDisabled, stickyHeaderEnabled (eventlogging will indicate which bucket the user is in by the group key).
- For all groups, an init event will be logged on page load (no other time) like the following:
QA Results - Beta
| AC | Status | Details |
|---|---|---|
| 1 | ✅ | T292587#7510832 |
