Document the relaunched Portal A/B test on https://meta.wikimedia.org/wiki/Discovery/Testing as soon as the implementation is complete.
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • debt | T112172 EPIC: of epics Wikipedia.org Portal UX tests to run | |||
| Resolved | • debt | T112209 EPIC: [Portal A/B/C test 2]: Make search type-ahead better with image and wikidata description on Wikipedia.org | |||
| Resolved | Ironholds | T122039 Document relaunched Portal A/B test |
Event Timeline
@Jdrewniak for this task I'm going to need some info on how the implementation actually ended up; what are the restrictions on getting into the schema? What are the restrictions on getting into the test? Where does the code live in case I need to eyeball it?
@Ironholds sure here's an overview:
Test Structure
Only user with Javascript enabled and relatively modern browsers will considered for event logging. ( IE8 and up ).
1 in 200 of these visitors to the page will be logged. This is the entire logging population.
70% of the logging population will be given the default experience and logged as a baseline.
30% of the logging population will be considered for the A/B/C test.
The testing population will differ from the baseline population because it is more restrictive. Users with browsers older than IE9 will be rejected.
The users in the A/B/C test will fall into 3 equal groups (10% each):
- Test1 - Described here: https://phabricator.wikimedia.org/T120269
- Test2 - Test with type-ahead suggestions (described below)
- control - this group will be given the same experience as the baseline, but the will reflect the more restrictive browser requirements of the A/B/C test population.
Second A/B test
The second A/B test builds on the first. It includes the changes of the first A/B test, but adds an improved type-ahead with search suggestions to the improved search-bar. Mockups for the type-ahead are available here:
https://meta.wikimedia.org/wiki/Wikipedia.org_Portal_Improvements?redirect=no
The code for the second A/B test is available in this repository:
https://phabricator.wikimedia.org/diffusion/WPOR/history/master/
Initial documentation at https://meta.wikimedia.org/wiki/Discovery/Testing#Portal_Search_Box_Tests