Page MenuHomePhabricator

Investigate why visual editor users have a higher save rate in EventLogging data but didn't in the A/B test
Closed, DeclinedPublic8 Estimated Story Points

Description

Baseline rates

From the edit analysis dashboard, on 7 August 2015 (using a 15-day moving average).

First edit save rate

WikiVisual editorWikitext editorVE advantage
itwiki0.2970.124140%
ruwiki0.2080.12172%
frwiki0.3360.18681%

1-5 edits experience

WikiVisual editorWikitext editorVE advantage
itwiki0.4000.35014%
ruwiki0.3590.25242%
frwiki0.4600.32641%

A/B test rates

This filters out sessions where the user aborted without making any changes to the page or aborted to switch from the visual editor to the wikitext editor. I think it also counts only the first five sessions of each users to remove outliers.

groupintra-edit sessionssuccessful savessuccessful save rate
control467729770.637
experimental424524490.577

Event Timeline

nshahquinn-wmf claimed this task.
nshahquinn-wmf raised the priority of this task from to High.
nshahquinn-wmf updated the task description. (Show Details)
nshahquinn-wmf added a subscriber: nshahquinn-wmf.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 6 2015, 8:35 PM
nshahquinn-wmf set Security to None.
Elitre added a subscriber: Elitre.Sep 3 2015, 1:41 PM
Halfak added a subscriber: Halfak.Sep 3 2015, 2:07 PM

Are you taking these measurements based on the staged deployment to newcomers? That would mean that my propensity hypothesis might not apply. I think that the Right(TM) next step is to compare like-to-like. Either use the edit analysis dashboard's metric of success (only counts successful saves -- not attempts. Also doesn't control for copy-paste edit sessions) to look at the A/B test user or apply the A/B test metric to recent registrations. See my methods here: https://meta.wikimedia.org/wiki/Research_talk:VisualEditor%27s_effect_on_newly_registered_editors/Work_log/2015-05-28#Edit_session_stats

Elitre awarded a token.Sep 3 2015, 2:09 PM
nshahquinn-wmf lowered the priority of this task from High to Medium.Sep 29 2015, 5:49 PM
Alsee added a subscriber: Alsee.Oct 15 2015, 10:12 AM

What you're seeing on the edit analysis dashboard is that editors using wikitext open a lot of throwaway edit windows strictly to view/learn/copy wikitext. Those extra disposable wikitext views make the percentages look low.
The A/B save rate shows that once someone has started making changes, the control (wikitext) group have a higher rate of successfully saving that revision.

The final A/B result was no measurable change in the number of people who successfully make initial edits, no measurable change in short or medium term retention, no measurable change in total contributions.

Opening the editor to copy and paste is a known workflow in VE as well.
This task is important because it may show what happens when you also include people who know how to use VE in the picture.
(There was also a nice point related to the time to save that Matt made.)

Alsee added a comment.Nov 5 2015, 11:26 PM

Opening the editor to copy and paste is a known workflow in VE as well.

Sure VE some windows may be opened like that, but the data indicates that in actual use the view-only windows being opened are predominantly wikitext. I haven't used VE much, but when I did, the support windows I opened were virtually all wikitext windows.

The original question: Investigate why visual editor users have a higher save rate in EventLogging data but didn't in the A/B test

Because the A/B test filtered out view-only windows, whereas EventLogging includes view-only windows.

To examine edit completion rates, I measured the proportion of intra-edit sessions that resulted in a "saveAttempt". I was careful to filter out intra-edit sessions where the user made no change before exiting (action.abort.type = "nochange") – which I expect is common for editors using wikitext editor to copy-paste some markup – or where the user was simply switching between editors (action.abort.type in ("switchwith", "switchwithout")). -- Halfak (WMF)

It not a "save failure" when extra view-only support windows are closed. What we've learned is that disposable workflow-support windows are predominately wikitext windows, and that raw EventLogging data is not an accurate metric for successfully completing edits.

Halfak (WMF) posted an excellent analysis of the May 2015 study data. The general outcome was that deploying VE had essentially no effect. Users given VE has a statistically-significant but negligibly lower revert rate, those given VE had notably slower editing, those given VE had a notably higher rate of failing to complete edits, but in the end no change in how many people managed to make their first edit, no change in retention, and no change in total contributions.

Zppix moved this task from Unsorted to Working on on the Contributors-Team board.Apr 26 2016, 2:31 PM
Deskana removed nshahquinn-wmf as the assignee of this task.Oct 31 2017, 3:03 PM
Deskana moved this task from Backlog to Freezer on the Contributors-Analysis board.
nshahquinn-wmf lowered the priority of this task from Medium to Low.Aug 2 2018, 8:07 PM
Restricted Application edited projects, added Product-Analytics; removed Product-Analytics (Kanban). · View Herald TranscriptOct 16 2019, 5:48 PM
nshahquinn-wmf closed this task as Declined.May 18 2020, 4:36 PM