We need a description of:
- How the sampling is working;
- What rate the sampling is at;
- What we are specifically tracking.
It can be in a phabricator comment, that's fine, I'll just stick it in the reports.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • debt | T112172 EPIC: of epics Wikipedia.org Portal UX tests to run | |||
| Resolved | • debt | T112173 EPIC: [Portal A/B test 1]: Change the size of search text field and search button on Wikipedia.org | |||
| Resolved | None | T120269 [PAB1] Describe upcoming portal A/B test |
Event Timeline
Comment Actions
For the first A/B test: T112173
1. How the sampling is working
Users who disable JS are ignored (EL limitation).
This is an EL limitation. EL relies on JavaScript.
Users who do not have localStorage in their browser are rejected.
LocalStorage is necessary to store a session ID and a last visit timestamp, so we can better track a same user activity.
Users are assigned in a group for 15 minutes.
They are given a new ID and assigned to a new group after 15 mins of inactivity.
They are extended for 15 more minutes otherwise.
This means every time you access the page:
if (now - lastTime) < 15min, extend session for 15 min.
if (now - lastTime) > 15min, (they are considered different users), so new ID, re-assign group and new session of 15 min.
Users who are running ie8 or below are rejected.
Writing back-compatibility JS is a pain when you can't use jQuery or similar.
To speed up the development and the A/B test release, and considering IE8 population is not so huge (I don't have the numbers with me though)
2. Sampling rate
Users that have JS, localStorage, and do not use ie8 or below:
- 199:200 are rejected
- 1:200 are tested:
- 1:2 is assign to the control group (baseline): this means 1:400 of those users are in control group.
- 1:2 is assign to the test group (a/b test): this means 1:400 of those users are in test group.