Background
We need to setup an A/B test to allow relevant mobile users to see recommendations. Thankfully for this feature, given that it is initiated via user interaction, we should be able to set this up via javascript on the client.
context: https://wikimedia.slack.com/archives/G8QAPHCTT/p1731615045692519
https://www.mediawiki.org/wiki/Readers/Web/Instrumentation_Overview
User story
- As an engineer on Web, I want to be able to show mobile recommendations to treatment users, and keep the status quo for control users
Requirements
- Should be fully revert-able with no impact to current search
- Should use generic solutions where it makes sense, and write our own code when needed
- WikimediaEvents has webABTestEnrollment.js for firing an event to the event platform
- core has mediawiki.experiments.js to setup buckets based off of the config
- Should support two group (control/treatment) experiments rolled out to the same percentage
- That percentage should be configurable per-wiki, and will change over the course of the experiment
- Should ideally support URL query params to enable/disable for testing
Implementation details
RelatedArticles config
- New config to store the ab test enabled/disabled state - This will allows us to enable/disable the AB test on a per wiki basis
- New config to store bucketing percentages - This will allow us to adjust the bucketing, as a percentage of total users, on a per wiki basis.
- This config should be in a format that can be passed to mw.experiments.getBucket() see here for format.
RelatedArticles JS
- Load the given configuration on the client and pass it to mw.experiments.getBucket(). We use session ID as the bucketing key.
- Trigger the experimental UI based on the given bucket.
- If in test group trigger logging + new UI, if in control group trigger logging, if excluded do nothing.
- Save the bucket to localStorage, ensuring that the user is given a consistent experience during the experiment duration.
- Set an expiry on the localStorage key for the duration of the experiment.
- On subsequent pageviews, we enable the experiment based on the saved bucket value, not the session ID.
- Add an override to load the experiment via URL param. (If possible, disable logging in this state?)
BDD
Feature: A/B Test Bucketing for Mobile Search Recommendations
Scenario: Displaying related articles in search based on A/B test bucket
Given the user is on the mobile site in an incognito window
When the user clicks on the search bar
Then half the users should see related articles in the empty state
And the other half should see nothing
And an ABEnrollment event is logged in the network tab with the group field reflecting the assigned bucketTest Steps
Test Case 1: Verify A/B Test Bucketing for Mobile Search Recommendations
- Open an incognito window and navigate to https://en.m.wikipedia.beta.wmflabs.org/w/index.php.
- Click on the search bar to initiate the search interaction.
- AC1: Confirm that for half the users, related articles appear in the empty state and that for the other half of users, the search bar remains empty with no recommendations.
- Open the Network tab in Google Chrome DevTools (right-click and select Inspect, then go to the Network tab).
- Filter the network requests to find events related to ABEnrollment.
- AC2: Verify that the group field in the event data reflects the assigned bucket (control or treatment).
Design
- N/A
Acceptance criteria
- We have implemented the ability to separate control and treatment groups for this feature
- We have hooked into relevant instrumentation created via https://phabricator.wikimedia.org/T378109
Communication criteria - does this need an announcement or discussion?
- N/A
Rollback plan
- What is the rollback plan in production for this task if something goes wrong?
Critically, we should be able to revert fully to the control without impact to current search
QA Results - Beta
| AC | Status | Details |
|---|---|---|
| 1 | ✅ | T378115#10399301 |
| 2 | ✅ | T378115#10399301 |
This task was created by Version 1.2.0 of the Web team task template using phabulous
