Page MenuHomePhabricator
Create Task
Maniphest T419066

[Suggestion Mode] Report on leading indicators
Closed, ResolvedPublic
Actions

Description

≥2 weeks after starting of the Suggestion Check experiment (T404600), we will check on a set of leading indicators (outlined below).

We will use this ticket to scope and conduct this analysis.

Analysis timing

≥2 weeks after the experiment begins (T421189).

Decision(s) to be made

  • 1. What – if any – adjustments/investigations will we prioritize for us to be confident moving forward with evaluating the Suggestion Mode MVP's impact in T404600?
    • None. We ended up completing this analysis near the end of the experiment. Even so, this analysis did not surface any issues/improvements that we thought warranted being addressed as part of this experiment. Instead, we will run another experiment designed to increase the likelihood that newcomers will see a suggestion.
  • 2. More broadly, will we continue running the experiment as-is?
    • Yes.

Leading indicators

Metrics

Leading indicatorMetric(s) for EvaluationNotesConclusion
Newcomers and Junior Contributors are not encountering Edit SuggestionsEdit suggestion exposure rate: 1) Proportion of articles that have at least one suggestion available (estimate of opportunities available for editors to find suggestions), 2) Proportion of editing sessions where at least one suggestion is available to be shown (how frequently do newcomers have the potential to encounter them when they edit), 3)Of all editing sessions where at least one Edit Suggestion is available to be shown, the proportion of editing sessions where at least 1 Edit Suggestion is seen (how frequently are newcomers seeing an edit suggestion when they are available within the articles they are editing)Need to investigate if enough suggestions will be shown to editors in the treatment group to achieve a sufficient sample size for the analysis. Relevant instrumentation: suggestion-shown-[] and suggestion-seen-[]
Newcomers and Junior Contributors are not engaging with Edit SuggestionsSuggestion engagement rate: Proportion of contributors who see at least 1 Edit Suggestion and click to accept at least 1 suggestion during their session.
Newcomers and Junior Contributors are not understanding the feature.Edit abandonment rate: Proportion of editing sessions that are abandoned after at least one Edit Suggestion was seen. Review abandonment rate with and without changes made. Overall, by suggestion type, and by if someone has accepted ≥1 suggestion
Newcomers and Junior Contributors do not find edit suggestions relevant.Edit suggestion decline rate: Proportion of contributors who see at least 1 Edit Suggestion and click to decline at least 1 suggestion during their session. Or Proportion of editing sessions where at least 1 Edit Suggestion is shown but no suggestions are expanded. Overall and by suggestion_typeNote: Suggestion mode will be available by default to the treatment group. Mobile and Desktop UI will have different options. Can’t hide on mobile but can hide on desktop.
Edit Suggestions are causing disruptionRevert Rate and Block Rate 1) Proportion of people blocked after engaging with an edit suggestion and 2) Proportion of published edits where a user engaged with an Edit Suggestion and are reverted within 48hours

Done

  • "Conclusions" documented in this ticket | @MNeisler
  • Make and document paths forward for "Decisions to be made" | @ppelberg
  • Findings published on mediawiki.org | @ppelberg
    • Results will be published on mw.org via T404600.
  • Leading indicators shared with volunteers participating in a/b experiment | @Quiddity
    • Results will be published on mw.org via T404600.

Loose

  • Among edit sessions when someone is seeing the See suggestions button for the first time, what proportion of people engage with the button? Of the people who engage with the See suggestions button, what proportion of people elect to see the Suggestion the button is making people aware of and in what proportion of people elect NOT to see the Suggestion the button is making people aware, by way of tapping the X button that appears within it?
  • What proportion editing sessions is least one suggestion available to be shown?
  • How frequently will users open an editor and be shown at least one edit suggestion? via @MNeisler
  • How (if at all) does the abandonment rate vary between edit sessions in which the See suggestions button (T414518) is and is not shown within?
    • We wonder the above aware of the potential for some people to find this button distracting. See T414518#11683955 for context.

Related Objects
Search...

Event Timeline

ppelberg created this task.Mar 4 2026, 9:17 PM
ppelberg moved this task from Untriaged to Editing-Q4-2025-2026 on the Editing-team board.
ppelberg updated the task description. (Show Details)Mar 6 2026, 10:54 PM
ppelberg mentioned this in T414518: Suggestion Mode: Ensure first suggestion discoverability when entering Edit mode regardless of viewport position.
ppelberg updated the task description. (Show Details)Mar 10 2026, 10:53 PM
ppelberg updated the task description. (Show Details)Mar 11 2026, 4:44 PM
ppelberg mentioned this in T418390: Determining treatment group exposure rates for Suggestion Mode MVP.Mar 11 2026, 4:47 PM
ppelberg updated the task description. (Show Details)Mar 12 2026, 9:25 PM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg assigned this task to MNeisler.Apr 16 2026, 10:05 PM
ppelberg triaged this task as High priority.
ppelberg updated the task description. (Show Details)
ppelberg edited projects, added: Product-Analytics, Editing-team (Editing-Q4-11May-22May-2026); removed: Editing-team.
ppelberg moved this task from Decision to be made to Release Prep on the Editing-team (Editing-Q4-11May-22May-2026) board.
ppelberg removed a project: EditCheck.
ppelberg added a subscriber: Quiddity.
ppelberg moved this task from Backlog to Meta on the VisualEditor Suggestion Mode board.Apr 16 2026, 10:07 PM
MNeisler edited projects, added: Product-Analytics (Kanban); removed: Product-Analytics.Apr 29 2026, 1:43 PM
ppelberg added a comment.Jun 6 2026, 12:19 AM

Per what @MNeisler and I discussed offline this week (3 June 2026): at present, the effect of Suggestion Mode is being diluted because new(er) volunteers are only seeing suggestions in 27% of test group edit sessions. The remaining test edit sessions have an experience identical to those in the control group, which pulls the measured effect toward zero.

Accordingly, we'd like to scope this leading indicator analysis to the test group edit sessions in which new(er) volunteers are seeing ≥1 suggestion so that we can more accurately gauge the effect of the intervention.

VPuffetMichel edited projects, added: Editing-team (Editing-Q4-22Jun-3Jul-2026); removed: Editing-team (Editing-Q4-11May-22May-2026).Tue, Jun 23, 8:06 PM
VPuffetMichel moved this task from Decision to be made to Release Prep on the Editing-team (Editing-Q4-22Jun-3Jul-2026) board.
MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Wed, Jun 24, 3:19 PM
MNeisler added a comment.EditedFri, Jun 26, 4:31 AM

@ppelberg Please see a summary of the leading indicator results below.
One key takeway: Edit suggestions have a clear, positive impact on edit completion rate and edit quality for users who view them. However, because the exposure rate is so low, we are not able to see these impacts in the overall experiment metrics.

Edit Suggestion Engagement (Treatment Only)

These metrics evaluate user engagement with the Edit Suggestions presented in the treatment group (summarized from TestKitchen Automated Analytics Dashboard as of 25 June 2026):

MetricResultsConclusions
Edit Suggestion Shown Rate: Proportion of edit sessions where the user spent at least 2 seconds in the ready state and at least one edit suggestion was available to be shown to the user54.7% of editing sessionsHigh Availability. Relevant suggestions are shown (available) for a majority (54.7%) of all editing sessions.
Edit Suggestion Seen Rate: Proportion of editing sessions where at least one edit suggestion was available (shown) and where the user viewed (seen) at least one edit suggestion.27% of editing sessions where a suggestion was available. (14.7% of all treatment sessions)Significant Discovery Gap. Users only saw (clicked to view) a suggestion in 27% of the sessions where one was available. Zooming out to the entire experiment, an Edit Suggestion was seen in only 14.7% of all editing sessions in the treatment group. This means 85.3% of the treatment group effectively had an identical user experience to the control group, heavily diluting the feature's observable impact on overall experiment metrics.
Edit suggestion acceptance rate: Proportion of contributors who see at least 1 Edit Suggestion and click to accept at least 1 suggestion during their session.29.2% of usersWhen users view a suggestion, about a third (29.2%) of users click to accept it. This is higher than the decline rate and confirms that a decent proportion of users who view a suggestion find them relevant enough to engage with.
Edit suggestion decline rate: Proportion of contributors who see at least 1 edit suggestion and click to decline at least 1 edit suggestion during a session.25.9% of users25.9% of users declined at least one edit suggestion they viewed. However, this is lower than the decline rates we've observed for Edit Checks (~55% [See Paste Check and Tone Check as examples])
Experiment Results: Control vs Treatment (Only editing sessions where Edit Suggestion was seen)

Experiment results shown on the TestKitchen dashboard, measure the global Intent-to-Treat (ITT) impact, comparing the entire Treatment group (where suggestions were made available) against the Control group. This shows the overall impact we'd have if we deployed the feature with its current exposure rate.

However, because new(er) volunteers only actively viewed an edit suggestion in 14.7% of all treatment sessions (see Edit Suggestion Seen Rate results above), these experiment metrics are heavily diluted by the 85.3% of users who had an identical experience to the control group. This explains why the dashboard shows minimal variance between the control and treatment groups.

To understand if Edit Suggestions genuinely work for users who see and interact with them, we wanted to filter treatment group results to just editing sessions where at least one Edit Suggestion was seen. This was done to evaluate the "undiluted" behavioral value and relevance of the suggestions themselves. Those results are summarized below:

NOTE: The results reported below are subject to selection bias. Because both groups are initialized identically upon opening the editor, filtering the treatment group down exclusively to users who actively click to view a suggestion captures an inherently higher-intent cohort. This group skews toward motivated editors compared to the control baseline, which includes lower-intent users or users who accidentally clicked the editor. Although the scale of impact shown above is likely inflated due to selection bias, the magnitude of lifts still strongly indicates that the feature has a positive impact on user experience.
MetricControl ResultTreatment ResultRelative % ChangeConclusions
Constructive activation rate27.10%56.91%+110.00% (2.1x)We observed the highest impact on constructive activation rate. Newcomers who clicked to view an edit suggestion are 2 times more likely to complete at least one constructive edit on their first day. This impact was large enough to pull up our overall experiment metrics despite the low exposure rate, resulting in a 2% relative increase across all editing sessions in the experiment (with an 89.95% probability that the treatment group outperformed the control).
Constructive Edit rate78.89%84.01%+6.49%Users who saw an edit suggestion are more likely to complete a constructive edit, indicating that the feature is successfully guiding newe(er) volunteer to publish quality edits.
Constructive Edit rate (desktop)82.15%84.93%+3.38%--
Constructive edit rate (mobile web)75.00%82.00%+9.33%We observed the highest impact on mobile web, where the rate increased by +10%, bringing it closer aligned with desktop rates.
Edit abandonment rate (with changes)7.00%8.36%+19.43%Seeing an edit suggestion results in more users starting to make a change instead of just abandoning the editor with no changes. As a result, we've moved more curious users further along in the funnel, causing abandonment with changes at this stage to increase.
Edit abandonment rate (without changes)21.03%16.03%-23.78%We saw a signficant 23.8% decrease in users abandoning their edits without making any changes. The suggestions successfully motivated users who would have typically left the editor with no changes to start typing.
Edit Completion Rate45.54%65.01%+42.75%Once users view an edit suggestion, their likelihood of successfully publishing jumps dramatically both on desktop and mobile. A portion of these increases is driven by selection bias (see note below); however, the magnitude of the lift strongly indicates that the feature has a positive impact on edit completion rate.
Edit Completion Rate Mobile39.80%48.58%+22.06%--
Edit Completion Rate Desktop49.87%69.28%+38.92%--
Second Week Edit Retention Rate All9.89%9.92%+0.30%We've seen positive directional impacts on second week edit retention, particularly on mobile but do not have sufficient data to confirm statistical significance to confirm trends
Second Week Retention Rate Mobile10.50%13.09%+24.67%--
Second Week Retention Rate Desktop10.73%10.25%-4.48%--
Time Until First Change48,553.34 ms49,223.17 ms+1.38%--
medelius subscribed.Fri, Jun 26, 5:27 AM
ldelench_wmf awarded a token.Fri, Jun 26, 1:20 PM
ldelench_wmf moved this task from Backlog to WE1.7 on the OKR-Work (WE1 FY2025-26) board.
ppelberg moved this task from Release Prep to Ready for Sign off on the Editing-team (Editing-Q4-22Jun-3Jul-2026) board.Fri, Jun 26, 7:20 PM
TAndic awarded a token.Tue, Jun 30, 1:42 PM
ppelberg mentioned this in T430878: Publish Suggestion Mode controlled experiment findings on-wiki.Wed, Jul 1, 9:01 PM
VPuffetMichel edited projects, added: Editing-team (FY2026-27 Q1 sprint 1); removed: Editing-team (Editing-Q4-22Jun-3Jul-2026).Fri, Jul 3, 8:18 PM
VPuffetMichel moved this task from Decision to be made to Ready for Sign off on the Editing-team (FY2026-27 Q1 sprint 1) board.
ppelberg claimed this task.Mon, Jul 6, 3:50 PM
ppelberg updated the task description. (Show Details)Mon, Jul 13, 11:21 PM
ppelberg closed this task as Resolved.Mon, Jul 13, 11:23 PM
ppelberg added subscribers: SonjaPerry, kzimmerman, MMiller_WMF.

After further offline discussion with @kzimmerman, @MMiller_WMF, @MNeisler, and @SonjaPerry, we've concluded:

  1. These initial results are promising and
  2. We cannot draw meaningful conclusions from them alone about Suggestion Mode's efficacy

Further, we do not think that further analysis of these results is capable of overcoming the self-selection biases at play. Accordginly, we are going to proceed with plans to run another experiment designed to increase exposure. this work will happen as part of DE1.1 (T430712).

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits