Document relaunched Portal A/B test
Closed, ResolvedPublic2 Story Points

Description

Document the relaunched Portal A/B test on https://meta.wikimedia.org/wiki/Discovery/Testing as soon as the implementation is complete.

Ironholds added a subscriber: Ironholds.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 21 2015, 10:40 AM
Ironholds set Security to None.Dec 21 2015, 11:52 AM
Ironholds edited a custom field.
Deskana triaged this task as "Normal" priority.Dec 22 2015, 9:10 PM
Deskana added a subscriber: Deskana.

@Jdrewniak for this task I'm going to need some info on how the implementation actually ended up; what are the restrictions on getting into the schema? What are the restrictions on getting into the test? Where does the code live in case I need to eyeball it?

@Ironholds sure here's an overview:
Test Structure
Only user with Javascript enabled and relatively modern browsers will considered for event logging. ( IE8 and up ).

1 in 200 of these visitors to the page will be logged. This is the entire logging population.

70% of the logging population will be given the default experience and logged as a baseline.

30% of the logging population will be considered for the A/B/C test.
The testing population will differ from the baseline population because it is more restrictive. Users with browsers older than IE9 will be rejected.

The users in the A/B/C test will fall into 3 equal groups (10% each):

  • Test1 - Described here: https://phabricator.wikimedia.org/T120269
  • Test2 - Test with type-ahead suggestions (described below)
  • control - this group will be given the same experience as the baseline, but the will reflect the more restrictive browser requirements of the A/B/C test population.

Second A/B test
The second A/B test builds on the first. It includes the changes of the first A/B test, but adds an improved type-ahead with search suggestions to the improved search-bar. Mockups for the type-ahead are available here:
https://meta.wikimedia.org/wiki/Wikipedia.org_Portal_Improvements?redirect=no

The code for the second A/B test is available in this repository:
https://phabricator.wikimedia.org/diffusion/WPOR/history/master/

Deskana closed this task as "Resolved".Jan 21 2016, 1:35 AM