Page MenuHomePhabricator

[PAB1] Describe upcoming portal A/B test
Closed, ResolvedPublic1 Estimate Story Points

Description

We need a description of:

  1. How the sampling is working;
  2. What rate the sampling is at;
  3. What we are specifically tracking.

It can be in a phabricator comment, that's fine, I'll just stick it in the reports.

Event Timeline

Ironholds created this task.Dec 3 2015, 8:16 PM
Ironholds raised the priority of this task from to Needs Triage.
Ironholds updated the task description. (Show Details)
Ironholds added a subscriber: Ironholds.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 3 2015, 8:16 PM

For the first A/B test: T112173

1. How the sampling is working

Users who disable JS are ignored (EL limitation).

This is an EL limitation. EL relies on JavaScript.

Users who do not have localStorage in their browser are rejected.

LocalStorage is necessary to store a session ID and a last visit timestamp, so we can better track a same user activity.

Users are assigned in a group for 15 minutes.
They are given a new ID and assigned to a new group after 15 mins of inactivity.
They are extended for 15 more minutes otherwise.

This means every time you access the page:
if (now - lastTime) < 15min, extend session for 15 min.
if (now - lastTime) > 15min, (they are considered different users), so new ID, re-assign group and new session of 15 min.

Users who are running ie8 or below are rejected.

Writing back-compatibility JS is a pain when you can't use jQuery or similar.
To speed up the development and the A/B test release, and considering IE8 population is not so huge (I don't have the numbers with me though)

2. Sampling rate

Users that have JS, localStorage, and do not use ie8 or below:

  • 199:200 are rejected
  • 1:200 are tested:
    • 1:2 is assign to the control group (baseline): this means 1:400 of those users are in control group.
    • 1:2 is assign to the test group (a/b test): this means 1:400 of those users are in test group.

Awesome; thanks!

JGirault renamed this task from Describe upcoming portal A/B test to [PAB1] Describe upcoming portal A/B test.Dec 9 2015, 10:02 PM
JGirault set Security to None.
JGirault edited a custom field.
JGirault moved this task from Backlog to Done on the Discovery-Portal-Sprint board.
debt closed this task as Resolved.Jan 20 2016, 9:39 PM
debt triaged this task as Medium priority.
debt added a subscriber: debt.

after chatting with Oliver, this task is done and resolved.

debt moved this task from Done to Completed on the Discovery-Portal-Sprint board.Jan 26 2016, 12:33 AM