This task involves the work of defining and documenting how the Editing Team will evaluate the impact(s) of the Suggestion Mode MVP.
Project objectives
To be effective, the first iteration of Suggestion Mode (aka MVP) will need to compel brand new editors to publish edits they might not have made otherwise and for the experienced volunteers who are reviewing said edits to consider them constructive.
Hypothesis
If we present people who enter the mobile VisualEditor with immediately actionable edit suggestions, then the proportion of edit sessions that result in someone publishing a constructive edit will increase.
User experience
New editors will encounter / experience the Suggestion Mode MVP in the following ways...
Upon opening the visual editor (desktop and mobile), they will be made aware that actionable suggestions exist for specific changes they can consider making to improve the article they are now accessing in "editing mode."
Note: the team is still deciding about whether these suggestions will be shown immediately to newcomers or whether they will need take an explicit action (e.g. tap a button) in order for them to appear.
Each of these suggestions will:
- Be visually related to the specific content/area of the article they are an effort to improve
- Contain ≥1 call to action that, when tapped/pressed, will initiate a workflow that guides them to make the change necessary to improve the content referred to in "1."
All other aspects of the visual editor's existing editing interface and publishing workflow will largely remain the same.
Drafts
| Document | Status | Notes |
|---|---|---|
| DRAFT Measurement Plan Suggestion Mode | @ppelberg reviewing | |
| WE 1.7 Edit Suggestions Instrumentation Spec | @MNeisler to populate once review of measurement plan is complete | |
Metrics
KPI
Curiosities
Guardails
Done
- Measurement plan is drafted
- Measurement plan is published on-wiki
- Specification is created for what – if any - new instrumentation is needed to support measurement/experiment plan (Q2)
Notes
A loose list for now. @MNeisler and @ppelberg to finalize before deployment.
- How many unique editors initiate an edit session in which the Suggestion Mode CTA is visible?
- Of the unique editors who saw the Suggestion Mode CTA is ≥1 edit session, what proportion of them enabled the feature?
- Of the unique editors who enabled Suggestion Mode, what proportion of them saw ≥1 edit suggestion?
- Of the unique editors who saw ≥1 edit suggestion, what proportion engaged with ≥1 edit suggestion?
- Of the unique editors who engaged with ≥1 edit suggestion, what proportion published an edit that was suggested to them?
- In order to understand the secondary effect(s) of this feature, we could consider answering something like...
- "Of the communities that did NOT block suggestion mode being enabled for newcomers through a controlled experiment, what proportion of experienced active editors engaged (exact metric tbd) with the mode in the # weeks before experiment plans were shared on-wiki? | Context
- We might also include some kind of qualitative survey (maybe using the existing "Decline survey" infrastructure?)...
- Proportion of edit sessions that reach ready state and involve someone spending ≥2 seconds in the editing interface that end without them making any changes