Background
- This ticket is for deploying the AB test to all wikis an a TBD date (Wed Feb 26 probably).
Requirements
Verify sampling rate based on @jwang's sampling estimation
enwiki sampling: 50/50 bucketing, 100% sample
all other wiki sampling: 50/50 bucketing, 100% sample
TBD: Configuration to be provided. There will be a different configuration for English correct given size, and another configuration for other projects.
Requirement
Deploy the Empty Search A/B test to all wikis on the planned deployment date (TBD: likely Wed, Feb 26). Verify that the sampling rate follows the expected configuration:
• English Wikipedia (enwiki): 50% of users are bucketed into the experiment, 100% sample.
• All other wikis: 50% of users are bucketed into the experiment, 100% sample.
BDD
Feature: Deploy Empty Search A/B test to all wikis
Scenario: Verify sampling rate for English Wikipedia
Given the A/B test is deployed to English Wikipedia
When users perform a search
Then 50% of users should be bucketed into the experiment
Scenario: Verify sampling rate for all other wikis
Given the A/B test is deployed to all other wikis
When users perform a search
Then 50% of users should be bucketed into the experimentTest Steps
Test Case 1: Verify sampling rate for English Wikipedia
- Visit English Wikipedia (en.wikipedia.org) in incognito mode to ensure a fresh bucket assignment.
- Open the developer console and run:
mw.loader.using('mediawiki.api').then(function () {
new mw.Api().saveOption('eventlogging-display-console', '1');
});- Perform a search in the search bar.
- Observe the event logs and check for an experiment-related event indicating bucketed into the experiment or not.
- Repeat steps 3-4 multiple times (at least 20 attempts) to confirm the 50% bucketing rate.
- AC1: Confirm that enwiki has a 50% bucketed rate with 100% sampling.
Test Case 2: Verify sampling rate for other wikis
- Visit a non-English Wikipedia wiki in incognito mode (e.g., fr.wikipedia.org, de.wikipedia.org).
- Open the developer console and run:
mw.loader.using('mediawiki.api').then(function () {
new mw.Api().saveOption('eventlogging-display-console', '1');
});- Perform a search in the search bar.
- Observe the event logs and check for an experiment-related event indicating bucketed into the experiment or not.
- Repeat steps 3-4 multiple times (at least 20 attempts) to confirm the 50% bucketing rate.
- AC2: Confirm that all other wikis have a 50% bucketed rate with 100% sampling.
QA Results - Prod
| AC | Status | Details |
|---|---|---|
| 1 | ✅ | T386849#10615833 |
| 2 | ✅ | T386849#10615833 |
Sign off steps
- Make sure the deployment steps have been documented and the analyst knows which data to ignore as part of the A/B test due to bugs.
