User story & summary:
Project-specific user story:
- As a newcomer, I want to receive a well-timed and engaging notification that guides me toward my first edit, so I can quickly understand what to do next and feel encouraged to contribute. By receiving clear guidance at the right moment, I will be more likely to take my first editing step and continue participating in Wikipedia.
Task specific user story:
- As the Growth team's Product Manager, I want to plan, run, and interpret the "Get Started" experiment, so that we can understand how constructive activation between the treatment group and control group differs.
Background & research:
Hypothesis: If new accounts that have not yet edited receive a supportive notification* with a Suggested Edit recommendation within 24 hours of creating an account, then they will be more likely to activate constructively.
*An Echo notification and an email if the account has an associated email address.
Supporting Data & Insights:
- The “Get Started” notification has already been shown to increase newcomer editing when sent at 48 hours after account creation (1). This suggests that well-timed interventions can positively impact newcomer activation. By sending the notification earlier, we may further improve activation rates by reaching users while their interest is still high.
- Prior studies show that positive reinforcement, such as the "Thanks" feature, leads to increased editor engagement (2). This suggests that notifications framed as encouragement rather than just instructions may yield better results.
Experiment Basics:
- Pilot wikis: eswiki, arwiki
- A/B test: 50% control (no notifications), 50% treatment
- Metrics:
- Key metric: Constructive Activation (finding + statistical significance)
- Secondary metric finding: Revert Rate (nice to have)
- Secondary metric: Retention (finding + statistical significance if possible)
Task scope:
Create scoped back measurement plan.
Analyze the collected data from the "Get Started" Experiment to assess its impact on constructive activation on mobile.
Share draft for Product Analytics peer review, and PM review.
Share analysis / report publically.
Background
- Project page: Constructive activation experimentation
- Related epic: T392256: [EPIC] "Get Started" notifications
Current full-page editing experiences require too much context, patience, and trial and error for many newcomers to contribute constructively. To support a new generation of volunteers, we will increase the number and availability of smaller, structured, and more task-specific editing workflows (E.g. Edit Check and Structured Tasks). The Growth team will primarily focus on Structured Tasks, while working closely with the Editing team to ensure our work integrates well with Edit Check.
This project aims to address the following user problem: Getting started editing on Wikipedia is difficult and especially frustrating on mobile devices. I want the editing interface to provide the in-the-moment policy and technical guidance I need, so my initial efforts aren't reverted.
This project aims to achieve the following user outcome: As a new Wikipedia volunteer, I feel confident and enthusiastic about contributing to the Wikimedia movement by editing Wikipedia articles. The tools provided guide me step-by-step, limit distractions, and allow me to learn progressively so I can successfully contribute on my mobile device.
As part of the Growth team 2024/2025 Annual Plan, the Growth team will explore various ways to increase constructive activation on mobile. This is part of the Wikimedia Foundation 2024-2025 Annual Plan, specifically the Wiki Experiences 1.2 Key Result
Key Discussions
- Slack: we1-new-contributor-experiences-fy25-26
- Slack: #we1-2_fy25-26_steerco
- Slack: growth-team
Acceptance Criteria
- Draft a scoped back Measurement Plan (example of previous Positive Reinforcement measurement plan)
- Discuss scope of the experiment with Growth's PM
- Instrumentation discussion with tech lead
- Final QA of instrumentation before the experiment release
- Quick check to ensure we are collecting the necessary data after the experiment release; Review experiment and instrumentation for experiment analysis.
- Review Data Collection Guidelines and obtain/seek L3SC approval as needed
- Review Data Modeling Guidelines as needed
- Review Dashboard(ing) Guidelines as needed
- QA
- Code Review: user data gathering, edit data gathering, analysis/modeling by @MNeisler
- ✨Perform analysis✨
- Prepare Quarto report
- Report review
- Share draft analysis
- Review data Reporting guide and Guidelines to ensure compliance
- Review publishing reports guidelines
- Determine and add the appropriate license
- Enter data publication into the data publication log form registry
- Post notebooks to Gitlab
- Publish the Quarto report following the web publishing guidelines including the Gitlab repo link
- Share findings with Growth team and the Community (either via a summary in this task, or a MediaWiki page).
- Clear all interim tables and csv files per the Data Retention Guidelines