Page MenuHomePhabricator
Create Task
Maniphest T403843

Product Analytics: enwiki analysis of constructive edits pre & post release
Closed, ResolvedPublic
Actions

Description

User Story
As the WMF Growth Team, I want to measure the impact of the Add a Link structured task rollout on English Wikipedia, so that we can understand whether it influenced the volume of constructive edits made by newer volunteers (the WE1.1 Key Result).

As an experienced English Wikipedia editor, I want to assess the impact of the Add a Link structured task rollout, so that I can determine whether the additional burden on patrollers is justified.

Hypothesis:
Given we scale “Add a Link” to 100% of newer volunteers at English Wikipedia, then newcomer constructive activation and retention will improve, which will increase constructive edits made by newer volunteers.

Details

  • The analysis should use the methods defined in T389404: FY2025/26 WE 1.1: Establish baseline for constructive edits.
  • "Constructive edits by newer volunteers" are defined as:
    • Edits in the main namespace on Wikipedia.
    • Made by contributors with ≤100 cumulative edits (including logged-out users).
    • Not reverted within 48 hours of publication.
  • Note: "Add a Link" is not available to logged-out/IP editors or temporary accounts.
  • The Add a Link task was scaled from a 20% rollout to 100% on September 3, 2025, at ~16:30 UTC.

Related tasks
T395524: Add a link (Structured task): Increase rollout on English Wikipedia to 100%
T389404: FY2025/26 WE 1.1: Establish baseline for constructive edits

Acceptance Criteria:

  • Analyze constructive edits on English Wikipedia for ~1 month before and ~1 month after the September 3, 2025 rollout.
  • Identify whether there is a meaningful change in the constructive edits metric that informs WE1.1.
  • Provide a data visualization of the findings.
  • If possible, segment results by platform (Mobile vs. Desktop).
NOTE: the hope is that this is a fairly quick Product Analysis "check", a full report is not needed. We previously completed a full A/B test while partially rolled out at English Wikipedia: https://analytics.wikimedia.org/published/reports/growth/add_a_link_enwiki_ab_test_report.html

Related Objects
Search...

Event Timeline

KStoller-WMF created this task.Sep 5 2025, 4:52 PM
KStoller-WMF moved this task from Inbox to Needs Discussion on the Growth-Team board.
KStoller-WMF updated the task description. (Show Details)Sep 5 2025, 5:02 PM
KStoller-WMF updated the task description. (Show Details)Sep 5 2025, 7:00 PM
KStoller-WMF updated the task description. (Show Details)Sep 5 2025, 7:02 PM
ldelench_wmf added a project: OKR-Work.Sep 8 2025, 2:20 PM
ldelench_wmf subscribed.
ppelberg added a parent task: T392954: [FY25-26] WE 1.1: Increase constructive edits.Sep 9 2025, 9:06 PM
ppelberg subscribed.
MNeisler claimed this task.Oct 3 2025, 4:21 PM
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Oct 9 2025, 5:34 PM
KStoller-WMF moved this task from Needs Discussion to Backlog on the Growth-Team board.Oct 17 2025, 12:11 AM
KStoller-WMF triaged this task as High priority.Oct 17 2025, 10:58 PM
KStoller-WMF moved this task from Backlog to Current Quarter Backlog on the Growth-Team board.
MNeisler added a comment.EditedOct 23 2025, 4:19 PM

@KStoller-WMF
I completed an analysis reviewing changes in constructive edits on English Wikipedia for ~1 month before and ~1 month after the September 3, 2025 100% rollout of the Add a Link structured task.

Please see a summary of findings below and let me know if you have any questions. More are details also available in this notebook.

Summary of Findings

I ended up reviewing constructive edit rates for both newer editors including unregistered users and newer editors limited to registered editors only. While we did not observe any significant changes in constructive edit rate when unregistered users were included, we did detect a statistically significant 1% increase when limited to registered newer editors who had access to the Add a Link Structured Task. These results are summarized below.

Constructive Edit Rate at English Wikipedia for Registered Newer Editors[1]
[1]Registered newer editor is defined as registered users with 100 or fewer cumulative edits

Time PeriodNumber of constructive editsProportion of constructive edits
Pre-deployment14755382.9%
Post-deployment15172883.7%

constructive_edits_daily_enwiki.png (667×1 px, 101 KB)

Constructive Edit Rate at English Wikipedia for Registered Newer Editors by Platform

Experiment GroupNumber of constructive editsProportion of constructive edits
Desktop
Pre-deployment11296085.5%
Post-deployment11937185.8%
Mobile web
Pre-deployment3459375.4%
Post-deployment3235776.5%

constructive_edits_daily_enwiki_platform.png (657×1 px, 127 KB)

Percent Change
We observed a 1% increase (+0.8 percentage points) in the constructive edit rate for registered editors with 100 or fewer edits at English Wikipedia following the Add a Link structured task rollout on 3 September.

Constructive edit increases were observed on each platform, with a slightly higher increase observed on mobile web. There was a 0.4% increase on desktop and a 1.5% increase on mobile web.

Is this change meaningful?

  • While small, results from a paired daily t-test indicate that this increase in constructive edit rate for registered users after the full rollout is statistically significant.
    • The calculated confidence interval confirms that we can be 90% confident that the true average improvement in the daily constructive edit rate is at least 0.26 percentage points.
  • This assumes that no other major external factors occurred between the two periods that could have caused the change. However, as this is not a controlled experiment, there may be external confounders that are impacting some of these changes.

Methodology

I reviewed all edits published on English Wikipedia one before and one month after the 3 September 2025 rollout that met the following requirements:

  • Completed in the main namespace
  • Made by contributors with ≤100 cumulative edits (including logged-out users).
    • Included analysis limited to registered users as well, since the Add a Link structured task is not available to logged-out editors.
  • Not reverted within 48 hours of publication.
MNeisler moved this task from Doing to Needs Review on the Product-Analytics (Kanban) board.Oct 23 2025, 4:27 PM
KStoller-WMF closed this task as Resolved.Oct 24 2025, 12:03 AM

Thank you, Megan!
Given the fact that unregistered users don’t have access to the task, I think it makes sense to limit to registered users like you did.

From my perspective this task is complete, we should just make sure to paste the results in the hypothesis report as well.

Urbanecm_WMF added a project: GrowthExperiments-NewcomerTasks.Wed, Jan 21, 2:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits