Page MenuHomePhabricator

Document relaunched Portal A/B test
Closed, ResolvedPublic2 Estimated Story Points

Description

Document the relaunched Portal A/B test on https://meta.wikimedia.org/wiki/Discovery/Testing as soon as the implementation is complete.

Event Timeline

Ironholds raised the priority of this task from to Needs Triage.
Ironholds updated the task description. (Show Details)
Ironholds subscribed.
Ironholds edited a custom field.
Deskana subscribed.

@Jdrewniak for this task I'm going to need some info on how the implementation actually ended up; what are the restrictions on getting into the schema? What are the restrictions on getting into the test? Where does the code live in case I need to eyeball it?

@Ironholds sure here's an overview:
Test Structure
Only user with Javascript enabled and relatively modern browsers will considered for event logging. ( IE8 and up ).

1 in 200 of these visitors to the page will be logged. This is the entire logging population.

70% of the logging population will be given the default experience and logged as a baseline.

30% of the logging population will be considered for the A/B/C test.
The testing population will differ from the baseline population because it is more restrictive. Users with browsers older than IE9 will be rejected.

The users in the A/B/C test will fall into 3 equal groups (10% each):

  • Test1 - Described here: https://phabricator.wikimedia.org/T120269
  • Test2 - Test with type-ahead suggestions (described below)
  • control - this group will be given the same experience as the baseline, but the will reflect the more restrictive browser requirements of the A/B/C test population.

Second A/B test
The second A/B test builds on the first. It includes the changes of the first A/B test, but adds an improved type-ahead with search suggestions to the improved search-bar. Mockups for the type-ahead are available here:
https://meta.wikimedia.org/wiki/Wikipedia.org_Portal_Improvements?redirect=no

The code for the second A/B test is available in this repository:
https://phabricator.wikimedia.org/diffusion/WPOR/history/master/