Page MenuHomePhabricator

Mentorship + Personalized Praise: Analysis
Closed, ResolvedPublic

Description

Analysis task

Spanish Wikipedia has Growth's Mentorship released to 50% of newcomers. English Wikipedia also has Mentorship released to only a percentage of new accounts created.
This gives us a unique opportunity to test the impact of mentorship.

Mentorship questions:
Are there any differences in activation between newcomers who receive mentorship and newcomers who don't? Does this differ if we only look at mainspace article editing vs. any namespace edit?
For editors who have already edited once, is there any differences in retention between newcomers who receive mentorship and newcomers who don't?
Are there any differences in revert rates between newcomers who receive mentorship and newcomers who don't?

Personalized praise questions:
Are mentees who are considered "praiseworthy" and have a mentor (and therefore might receive a personalize praise message) more likely to be retained than a mentee that is praiseworthy but does not have a mentor (and therefore not receiving praise).
Does a mentee's engagement (edit count) increase after receiving personalized praise from their mentor, compared to a similar editor who does not receive a message?

Background

Research shows that praise and encouragement from other users increases newcomer retention.
The personalized praise idea is part of a larger Growth team Positive Reinforcement effort.

Acceptance Criteria
  • Provide a short written overview detailing methodology and answers to the questions detailed in this task.
  • Review results with Growth PM and Growth team

Event Timeline

KStoller-WMF updated the task description. (Show Details)
nettrom_WMF raised the priority of this task from High to Needs Triage.Jul 7 2023, 9:18 PM

I've added this to Product Analytics tag and reset the priority so the PA team can triage it there. Given the timeliness of this work in preparation for Wikimania, I think the High priority should be re-added at triage. I'll then also move it to the PA kanban board to pick it up next week.

I've completed a preliminary analysis of mentorship and made a writeup of the findings on mw:Growth/Positive reinforcement/Mentorship preliminary analysis, August 2023. The goal of this analysis was to provide some initial insights into mentorship in support of the upcoming Wikimania discussion. We therefore focus on high-level questions such as what proportion of newcomers ask questions, how quickly they do, and to understand if mentorship has an impact on the Growth KPIs. The writeup is therefore also fairly high-level and without specifics for now, but does have some supporting visualizations.

nettrom_WMF lowered the priority of this task from High to Medium.Aug 24 2023, 10:59 PM

Reducing the priority on this to Medium since we've completed the time-sensitive part of this.

nettrom_WMF moved this task from In Progress to Prioritized on the GrowthExperiments-Mentorship board.
nettrom_WMF moved this task from Doing to Next 2 weeks on the Product-Analytics (Kanban) board.

Moving this out of the various doing/in progress columns as I'll instead be focusing on Levelling Up for a bit.

The notebooks for the previously mentioned preliminary analysis of the effects of mentorship have been added to this Gitlab repository using a naming convention that reflects that they're associated with this phab task. Resolving this parent task now, and I'll close the child tasks once their notebooks are published as well.