== Background
We would like to run an A/B test on the page issues feature for two weeks. The test will display the new page issues code to one group and the older version to the remainder (as per the bucketing code built in {T193584} and the instrumentation built in {T191532}. We will collect data for 4 weeks, plus a run-up time of 1-2 days to limit caching and novelty effects. A separate task has been set up for turning the test off: T200793 . Suggested sampling ratio: 20% (per T200792#4489268)
== Questions we are trying to answer
See {T200794}
== Acceptance criteria
[x] Notify the Analytics Engineering team as described [[https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Data_retention_and_auto-purging#Black-listed_schemas |here]] to have both ReadingDepth and PageIssues blacklisted from being stored in MariaDB (see T200792#4489268).
(done, see corresponding AC in T191532)
[-] Enable the A/B test on a single wiki (Latvian Wikipedia, e.g.https://lv.wikipedia.org/wiki/Ropa%C5%BEu_sporta_centrs) at a sampling ratio of 100%: T204609
[-] Run for 2-3 days and check the data coming in (TBC by Olga) - checks made on Latvian wiki - https://phabricator.wikimedia.org/T204609#4630216
[-] Ensure wikis are ready for testing
Japanese: https://phabricator.wikimedia.org/T204090#4626354
Russian: https://phabricator.wikimedia.org/T204090#4626381
Persian: https://phabricator.wikimedia.org/T204090#4626510
[-] Roll out using several deploy windows in a single day - take care rolling this out in case there are any unexpected spikes in errors or event logging:
[-] Progressively roll out page issues to the target projects at 5%
[-] Progressively roll out page issues to the target projects at 10%
[-] Set A/B test for page issues for select projects, at the A/B test sampling ratio of 20% for enwiki (per T200792#4489268) and 100% for the other three, smaller wikis (per T200792#4632005 ).
[-] Increase sampling rate on jawiki, ruwiki, fawiki to 100%
Tentative dates:
- Latvian wiki: sept 19
- English, Russian, Japanese, Farsi: oct 1 (Switch from Catalan to Farsi due to some smaller issues on Catalan and so that we can also have an rtl language included in the test)
== Sign off steps
[x] Wait a few days to get confirmation from Tilman that the A/B test instrumentation working as expected.
[x] If necessary, create follow-up tasks (for bugs, inconsistencies, etc)
--> T204143 in particular
[x] Setup task to analyse A-B test
--> T200794
[] Prepare deploy/remove code task that will be carried out based on A-B test results.
[] Add a note to the project page
[] Add a note to the [release timeline](https://www.mediawiki.org/wiki/Reading/Web/Release_timeline)
== TODO
1. Use @phuedx or @pmiazga's "bucket breaker" script to find a session ID that'll put you in the correct bucket for testing
- https://gist.github.com/phuedx/d580f01c501d207398828b717bf9870b
Run this code to opt into the PageIssues A/B test:
```
M = mw.mobileFrontend;
AB = M.require( 'skins.minerva.scripts/AB' );
var t = 0, abTest;
function check() {
abTest = new AB( {
testName: 'WME.PageIssuesAB',
// Run AB only on article namespace, otherwise set samplingRate to 0,
// forcing user into control (i.e. ignored/not logged) group.
samplingRate: mw.config.get( 'wgMinervaABSamplingRate', 0 ),
sessionId: t
} );
}
check();
while (!abTest.isB()) {
t+=1;
check();
}
mw.storage.session.set('mwuser-sessionId',t);
```