Page MenuHomePhabricator

Variant tests: A/B/C testing capability
Open, MediumPublic

Description

Unlike previous interventions by the Growth Team, we do not plan to run Newcomer Tasks as a straightforward A/B-test to see if it improves activation and retention. This is partly because we see this feature as an incremental change to the Homepage itself, and we have already been running an A/B-test with the Homepage to determine its effect on activation and retention. It is also because we will be able to learn how to improve newcomer tasks if we can test multiple variants of the feature.

Therefore, for Newcomer Tasks we plan to give the intervention to a large proportion of new users and keep a small control group, in what we will refer to as an A/B/C test. In this configuration the A and B groups will both get the Newcomer Tasks intervention, but different variants of it (some possible variants are described below). The C group is our control group, which will not have access to the Homepage by default. The proposed distribution of these groups is 40% / 40% / 20%.

This task is to make it possible to experiment with multiple variants simultaneously, and to record and manage which users are receiving which variants in order to facilitate analysis.

The specifications for how we want to run these tests is listed in this section of the newcomer tasks measurement plan.

Event Timeline

LGoto triaged this task as Medium priority.Oct 21 2019, 5:04 PM

We've decided that A/B/C testing is not required for v1.0, but that we do want to have it before v1.1.

MMiller_WMF moved this task from Inbox to Upcoming Work on the Growth-Team board.
MMiller_WMF renamed this task from Newcomer tasks: A/B/C testing capability to Variant tests: A/B/C testing capability.Nov 22 2019, 1:02 AM
MMiller_WMF updated the task description. (Show Details)Nov 26 2019, 12:33 AM

Note: for our initial simple implementation of variant testing (for instance, to run the "initiation" test), there may not be any special work to be done with this task. If that ends up being the case, we can put this task in our backlog, and make it about any work we want to do that would make our variant tests more sophisticated.

We don't need to do any special work for this task, and T238888: Variant tests: "initiation" test (A vs. B) is already in code review, so I think this task can go into the backlog.

nettrom_WMF moved this task from Triage to Backlog on the Product-Analytics board.