Page MenuHomePhabricator

Positive Reinforcement: evaluate Leveling Up leading indicators
Closed, ResolvedPublic


User story:
As the Growth team, I want to quickly determine if the new Positive Reinforcement features are having a positive impact on newcomers, because if we see any negative impact then we will want to revert changes and if we don't see any positive changes then we will need to make further improvements or consider sunsetting the features.

The Positive Reinforcement features aim to provide or improve the tools available to newcomers and mentors in three specific areas that will be described in more detail below. Our hypothesis is that once a newcomer has made a contribution (say by making a structured task edit), these features will help create a positive feedback cycle that increases newcomer motivation.

Experiment plan
Similarly as we have done for previous Growth team projects, we want to test our hypotheses through controlled experiments (also called "A/B tests"). This will allow us to establish a causal relationship (e.g. "The Leveling Up features cause an increase in retention of xx%"), and it will allow us to detect smaller effects than if we were to give it to everyone and analyze the effects pre/post deployment.

In this controlled experiment, a randomly selected half of users will get access to Positive Reinforcement features (the "treatment" group), and the other randomly selected half will instead get the current (September 2022) Growth feature experience (the "control" group). In previous experiments, the control group has not gotten access to the Growth features. The team has decided to move away from that (T320876), which means that the current set of features is the new baseline for a control group.

These experiments will first run on the pilot wikis. We can extend this to additional wikis if we find a need to do that, but it would only happen after we have analyzed the leading indicators and found no concerns.

Each experiment will run for approximately one month, and for each experiment we will have an accompanying set of leading indicators that we will analyze two weeks after deployment.

Leveling up: treatment group gets both the updated Impact module and the Leveling up features.

Acceptance Criteria:

Event Timeline

Moving this to the Product Analytics kanban board and the Growth Team's current sprint board as this work will need to start either this week or next.

KStoller-WMF raised the priority of this task from Medium to High.Apr 2 2023, 8:10 PM

The leading indicators have been defined and I've created a notebook to aggregate data and set it up to run once a day. We also have a Superset dashboard to monitor the indicators. What remains is:

  • Add statistics for the two notifications since they are now deployed.
  • Add the leading indicators and a snapshot of the statistics to the project page on MediaWiki, together with some discussion about what the statistics mean.
  • Follow up on the slight drop in constructive activation on mobile for users who get the New Impact module.

I'm creating a child task for the last point.

nettrom_WMF lowered the priority of this task from High to Medium.Apr 10 2023, 4:20 PM

Reducing the priority of this to Medium as the majority of this work is done and not as urgent. I'll also remove the due date.

The findings for all indicators have been posted on wiki here: Closing this task as resolved.