User stories:
As a Growth team member, I want to understand the impact of the new Positive Reinforcement features on newcomers, because that's how we will measure the success or failure of these features.
Related tasks:
T328757: Leveling up: Define feature flag for gating the functionality
Measurement plans:
We want to test our hypotheses through controlled experiments (also called “A/B tests”). This will allow us to establish a causal relationship (e.g. “The Levelling Up features cause an increase in retention of xx%”), and it will allow us to detect smaller effects than if we were to give it to everyone and analyse the effects pre/post deployment.
In this controlled experiment, a randomly selected half of users will get access to Positive Reinforcement features (the “treatment” group), and the other randomly selected half will instead get the current (September 2022) Growth feature experience (the “control” group). In previous experiments, the control group has not gotten access to the Growth features. The team has decided to move away from that (T320876), which means that the current set of features is the new baseline for a control group.
Full measurement specification.
Acceptance Criteria:
For Growth pilot wikis (ar, bn, cs, and es Wikipedia):
- 50% of new user registrations receive Positive reinforcement features (new impact module & leveling up), 50% will not receive any Positive reinforcement features.
- Existing users with Growth features enabled will receive all Positive Reinforcement features
- Autocreated accounts are excluded from this test.