Page MenuHomePhabricator

Document relaunched Portal A/B test
Closed, ResolvedPublic2 Estimate Story Points


Document the relaunched Portal A/B test on as soon as the implementation is complete.

Event Timeline

Ironholds raised the priority of this task from to Needs Triage.
Ironholds updated the task description. (Show Details)
Ironholds added a subscriber: Ironholds.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 21 2015, 10:40 AM
Ironholds set Security to None.Dec 21 2015, 11:52 AM
Ironholds edited a custom field.
Deskana triaged this task as Medium priority.Dec 22 2015, 9:10 PM
Deskana added a subscriber: Deskana.

@Jdrewniak for this task I'm going to need some info on how the implementation actually ended up; what are the restrictions on getting into the schema? What are the restrictions on getting into the test? Where does the code live in case I need to eyeball it?

@Ironholds sure here's an overview:
Test Structure
Only user with Javascript enabled and relatively modern browsers will considered for event logging. ( IE8 and up ).

1 in 200 of these visitors to the page will be logged. This is the entire logging population.

70% of the logging population will be given the default experience and logged as a baseline.

30% of the logging population will be considered for the A/B/C test.
The testing population will differ from the baseline population because it is more restrictive. Users with browsers older than IE9 will be rejected.

The users in the A/B/C test will fall into 3 equal groups (10% each):

  • Test1 - Described here:
  • Test2 - Test with type-ahead suggestions (described below)
  • control - this group will be given the same experience as the baseline, but the will reflect the more restrictive browser requirements of the A/B/C test population.

Second A/B test
The second A/B test builds on the first. It includes the changes of the first A/B test, but adds an improved type-ahead with search suggestions to the improved search-bar. Mockups for the type-ahead are available here:

The code for the second A/B test is available in this repository:

Deskana closed this task as Resolved.Jan 21 2016, 1:35 AM