This task is about re-analyzing the results of the mobile VE as default A/B test. [i]
Decision to be made
The decision this analysis is intended to help us make:
What editing interface should be shown to people who do not have a preference already set?
Decision points
These are the metrics we intended to calculate as part of this analysis:
| Scenario | Title | Scenario | Action |
|---|---|---|---|
| 1. | People who are shown the visual editor by default are more likely to continue editing | default-visual has a higher retention rate than default-source | 1. Confirm revert rate meets current levels; 2. Confirm ready-rate is not significantly lower than other editing interface; 3. Create a proposal to make the VisualEditor the default mobile editing interface on all wikis; 4A. For wikis where showing newcomers and logged out editors the visual editor by default on mobile would be different from what is done on desktop (e.g. en.wiki), we would treat the A/B test results as the beginning of a conversation with that wiki. The purpose of that conversation would be to decide, together, how the mobile editing interfaces should be configured for newcomers. 4B. For wikis where showing newcomers and logged out editors the visual editor by default on mobile would be the same as what is done on desktop (e.g. ar.wiki), we would inform these wikis of the results, share our plan to make the visual editor the default on mobile for newcomers and logged out users and invite people to share evidence they think should lead us to reconsider this plan of action. |
| 2. | People who are shown the source editor by default are more likely to continue editing | default-source has a higher retention rate than default-visual | Investigate what could have contributed to this outcome (reason: the empirical studies we've done to-date clearly show newcomers have more success and are more comfortable with the visual editor) |
| 3. | People who are shown the source editor by default and people who are shown the visual editor by default are equally likely to continue editing | default-source has a similar retention rate than default-visual | Compare interfaces across other test metrics: the total number of edits made, revert rate, edit completion rate, % of editors who make at least one successful edit |
Variables that will influence decision
The editing interface that is shown by default on mobile is the interface that:
| Affect | Metric(s) | Definition |
|---|---|---|
| Causes more people to continue editing Wikipedia | editor retention | 2nd week / 2nd month / 6 months |
| Causes more people to complete the edits they set out to make | edit completion rate and % of editors who make at least 1 successful edit | edit completion rate = the proportion of ready events that reach saveSuccess; successful edits = edits that reach SaveSuccess and are not reverted. |
| Does not cause more vandalism | Revert rate | The percentage of completed edits reverted within 48 hours [ii] |
Metrics
These are the metrics we intended to calculate as part of this analysis:
| Metric(s) | Definition |
|---|---|
| Editor retention | 2nd week / 2nd month / 6 months |
| Edit completion rate | the proportion of ready events that reach saveSuccess |
| % of editors who make at least 1 successful edit | % edits that reach SaveSuccess and are not reverted w/in 48 hours [ii] |
| Edit quality (read: revert rate) | The percentage of completed edits reverted within 48 hours [ii] |
| Disruption | The percentage of edits completed in an editing interface that is not the one they were shown by default |
| Total number of completed edits | Total number of edits that reach SaveSuccess and are not reverted w/in 48 hours [ii] |
| Load time | The amount of time that elapses between when someone presses edit (init) and the editor is ready (ready) |
Open questions
- 1. Considering the number of "exploratory" clicks, might it be better to define edit completion rate as: the proportion of firstChange events that reach saveSuccess?
- 2. How much higher does the revert rate in either mode need to be for it to be "significant"?
- See: T259196#6374218
- 3. Is 90 days worth of data sufficient to confidently make this decision?
i. https://www.mediawiki.org/wiki/VisualEditor_on_mobile/VE_mobile_default
ii. While I recall "48 hours" being the standard window within which an edit is considered to have been reverted or not, I have yet to find documentation that corroborates this.
Notes
- The AHT team will be implementing a change that could cause more anonymous editors to start editing on mobile. This change could lead to more anons starting editing sessions on mobile. We should consider this if we notice any changes in the A/B test results around anons.
- Details about this work can be found in T189717