Page MenuHomePhabricator

Leveling Up: post-experiment questions & analysis
Closed, ResolvedPublic

Description

User story:

  • As the Growth team, I want to understand how Leveling up (part of the Positive Reinforcement project) has changed newcomer behavior, because I want to see if this work was effective and I want to know if any improvements should be made.
  • As a Wikimedian, I want to make sure new Growth features are working well for newcomers, because then I'll be more accepting of the changes when they are rolled out on my wiki.

Positive Reinforcement experiment plan:
https://www.mediawiki.org/wiki/Growth/Positive_reinforcement#Experiment_plan


Follow up questions:

Post-edit notifications (“try new task type” dialog & “try suggested edit” dialog)

  • We know the click through rate, but what is the completion rate after click though?
  • Should we consider adjusting when we show the "try suggested edits" post-edit dialog? Is there evidence to suggest it would be more impactful at first or second edit, and we should remove the dialog after 7th edit?
  • Should we consider adjusting when we show the “try new task type” dialog?
  • What is the impact of the post-edit dialogs on Growth KPIs?
    • Activation (edit 1st 24h)
    • Retention — do they return within two weeks after being activated?
    • Productivity — how many edits made within 15 days?
    • Revert rate - how many edits are reverted?

Acceptance Criteria:

  • Answer follow-up questions that Growth team has based on initial analysis report
  • Discuss results with Growth team (in synchronous meeting or async)

Event Timeline

@nettrom_WMF - I added this task, but the list of questions isn't finalized. Let's discuss more at our next 1:1 and refine the list of questions.

In order to break up work into manageable sizes for sprints, I've moved the documentation part of this task into: T348896: Leveling up experiment analysis: Document significant findings on Wiki
And we can consider this task complete.