Page MenuHomePhabricator

Calculate revert rate baseline
Closed, ResolvedPublic

Description

This task is about calculating the desktop revert rate for newcomers (defined below) editing Wikipedia on desktop.

Background

It is possible people who are shown the visual editor on mobile and the people who are shown the source editor on mobile are equally likely to be retained (read: to start and continuing editing Wikipedia).

Should the scenario described above arise, we will look at other metrics – of which revert rate is one – to determine which editing interface should be shown to newcomers by default. Specifically, we will look at whether the revert rate in one test group (default-visual or default-source) is significantly higher than the comparable revert rate on desktop.

This task is about calculating said "comparable revert rate on desktop" which will serve as the baseline to which we will compare the test group revert rates.

Analysis

We are curious to learn the percentage of edits that are reverted on desktop that meeting the following conditions:

  • Edits made on desktop web
  • Edits made using the visual editor or the wikitext editor
  • Edits made by people (read: not bots)
  • Edits made by people who are logged out or who have made 0 cumulative edits
  • Edits made to a Wikipedia content namespace
  • Edits that are reverted within 48 hours of being made

Done

  • The "Analysis" described above is computed

Event Timeline

ppelberg updated the task description. (Show Details)Jul 29 2020, 10:19 PM
ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.
MNeisler triaged this task as Medium priority.Aug 3 2020, 3:00 PM
MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

Here are the baseline revert rates for anonymous users or first-time editors on Wikipedia desktop on each editing interface. In this analysis, revert rate is defined as the proportion of non-bot edits which were reverted within 48 hours.

Overall revert rate (2019-07 to 2020-07)
VisualEditor: 30.1%
Wikitext: 23.9%

There is a fairly consistent difference in monthly revert rates between the two editing interfaces over the past two years, with the maximum difference of around 8%. The average difference in monthly revert rates between the two editing interfaces for the past two years is 6.2%. Visual Editor has a higher revert rate ranging from 24% to 32% while wikitext's revert rate ranges from about 21% to 26%.

Based on these numbers, I think if we see a difference of revert rates any above 10% between the two test groups would be significant and should be considered along with the other identified metrics comparing the two groups. If we do find a significant difference in revert rates, I'd recommend comparing the distribution of revert rates and users within each test group to get more insight into the source of the reverts.

Superset Dashboard for Self-Exploratory Data
https://superset.wikimedia.org/r/295

You can use the filter box to review revert rates for each interface, language, if the editor is anonymous, or the editor's cumulative edit count bucket.
Note this data includes edits reverted anytime before the most recent snapshot (not within 48 hours) and can not filter to just editors with 0 cumulative historical edits so the rates will differ slightly from what is described above.

See notebook for further details and results.

ppelberg closed this task as Resolved.Aug 13 2020, 1:36 AM

What you shared in T259196#6374218 looks good – thank you, @MNeisler.

Per the conversation we had today, I'm going to resolve this task considering we now have the data it was asking for.

If further analysis is needed following the conversation we will be having with the team about the "revert rate significance threshold" [i], we can either re-open this task or file a new one.


i. From T221198's "Open questions" section:

  1. How much higher does the revert rate in either mode need to be for it to be "significant"?
Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptAug 13 2020, 1:36 AM