Page MenuHomePhabricator

New Impact Module: experiment analysis
Closed, ResolvedPublic


User story:

  • As the Growth team, I want to understand how the new Impact Module has changed newcomer behavior, because I want to see if this work was effective and I want to know if any improvements should be made.
  • As a Wikimedian, I want to make sure new Growth features are working well for newcomers, because then I'll be more accepting of the change when they are rolled out on my wiki.

Positive Reinforcement Experiment Plan:

Acceptance Criteria:

Event Timeline

Just noting that this initial experiment analysis is leading us to look into activation and the current Impact Module empty state further: T330614: New Impact module's empty state on mobile: research spike

The team noted a few questions that would help us decide on next steps:

  • Comparing the funnel of users who tap on empty impact module, then press "See suggested edits". (If there is a marked decrease in users who click on "See suggested edits" in the new impact module, that could help explain the lower activation rate.)
  • Can we correlate the activation data with the device information (ex. Mobile screen sizes)?
  • We are using ~6 weeks of data, and there have been several changes during that time. If we compare the first 3 weeks to the second 3 weeks, does the difference in activation rate between the experiment groups look about the same?

I added a short summary of Impact module results here:
Please feel free to make updates! My assumption is that we will share a much more detailed report on wiki for the Leveling up report and when we wrap up the Positive Reinforcement work, but I figured I should at least share the details we have now.

Should we consider this task resolved, or is there more you hope to do for impact module analysis?

@KStoller-WMF : I made some changes to the summary of the findings. There's now a graph for the key finding, and I elaborated on the work we've done to investigate and mitigate the issue. I'll close this task as resolved, but you should feel free to edit the results section or reach out to me with proposed changes as you see fit.