This tasks holds the work of running a controlled experiment to evaluate the impact of enabling people on mobile web to edit any article section, regardless of which edit icon they first tap (T387175).
Deployment timing
| Milestone | Target completion date | Ticket | Responsible | Status | Notes |
|---|---|---|---|---|---|
| Implement UX | ✅ | T409990 | @Esanders | DONE | Team will consider further refinements after usability testing is completed |
| Draft Tech/News announcement | ✅ | T409112 | @Quiddity | DONE | |
| Review Tech/News announcement | ✅ | T409112 | @ppelberg | DONE | |
| Complete usability testing (1st Round) | ✅ | T410614 | @bmartinezcalvo | DONE | Run first round of usability test; Key decision: what (if any) UX issues will we address before start of A/B experiment |
| Complete usability testing (2nd Round) | T410614 | @bmartinezcalvo | NOT NEEDED | This will only be run if/when we come to learn aspects of the initial test protocol or user experience needs revision | |
| Publish usability testing findings | ✅ | T410614 | @bmartinezcalvo | DONE | |
| Finalize UX refinements (if any) | ✅ | T410614 | @ppelberg | DONE | We've prioritized one refinement: T411669 |
| QA instrumentation and experiment set-up | ✅ | T410800 | Experiment platform | DONE | |
| Complete button QA | Monday, 8 Dec | T410319 | Editing QA + @MNeisler | In progress | |
| Implement instrumentation and experiment set-up | Tuesday, 9 Dec | T410800 | @MNeisler | Experiment end date NEEDED | |
| Conduct pre-deployment QA of UX | Wednesday, 10 Dec | T411669 | Editing QA | ||
| Start test | Thursday, 11 Dec | T409112 | Editing Engineering | ||
Tech/News
Later this week, a controlled experiment will begin for editors on the 100 largest Wikipedias who are editing a section in the mobile website Visual editor. 50% of editors will notice a new "Edit full page" button that will enable them to expand their editing session to the whole page. The experiment will last ~4 weeks. This feature is intended to make it easier for people on mobile web to edit any article section, regardless of which section-edit icon they tapped to begin. Please visit T409112 to see more details about the project.
UX
 |  |  |
| First section | Middle section | Last section |
Hypothesis
If we enable people on mobile web to edit any article section, regardless of which edit icon they first tap, then the newcomer mobile edit abandonment rate will decrease by 1% because they will be able to more more easily locate the content they tapped edit seeking to change.
Decision to be made
What – if any – adjustments to the mobile web {nav Edit] button UX need to be made before we can be confident all of the following are true?
- .
- .
KPIs
The main outcomes we are trying to impact through this feature. These are what we are primarily using for evaluating the hypothesis and deciding whether to deploy an intervention more widely.
| Hypothesis | Decision(s) to be made | Metric description |
|---|---|---|
| If we enable people on mobile web to edit any article section, regardless of which edit icon they first tap, then the newcomer mobile edit abandonment rate will decrease by #% because they will be able to more more easily locate the content they tapped edit seeking to change. | ||
Secondary metrics
Used to learn about additional impact of revising the mobile web editing button behavior and are not primary targets of the intervention. They reveal side effects (both positive and negative) of trying to improve the Primary Metric with the intervention.
| ID | Hypothesis | Metric description |
|---|---|---|
| Curiosity #1 | Constructive edits by newcomers will increase because more people acting in good faith will have an easier time locating the content they tapped edit seeking to change | Proportion of published edits[i] by users with ≤100 cumulative edits that are constructive [i] |
ii: "Constructive edits" = edits to pages in any Wikipedia main namespace that are not reverted within 48 hours of being published
Leading indicators
TICKET NEEDED
Guardrails
Used to make sure that the new checks presented are not negatively impacting an editor’s experience completing an edit or causing disruption on the wikis. The scenarios named in the chart below emerged through T325851.
| Guardrail Name | Metric description | Notes |
|---|---|---|
| Constructive edits decrease | By making it easier for people tapping edit on mobile web to change larger portions of the article, we may reduce the friction to publishing destructive edits | |
| Edit quality decrease | Proportion of published edits that add new content and are reverted within 48 hours. | |
| Edit completion rate drastically decreases | Proportion of edits started (defined as TBD) that are published. | |
| People are blocked at higher rates | ||
A/B Test: Decision Matrix
| ID | Scenario | Indicator(s) | Plan of Action |
|---|---|---|---|