Page MenuHomePhabricator

Investigate drop in mobile VE edit completion rate among unregistered users
Open, HighPublic

Description

In T236860#5624424, we learned around 4-July-2019 there was a drop in the edit completion rate among unregistered users attempting to edit using mobile VE.

This task is intended to represent the work involved with uncovering potential causes for this drop.

Event Timeline

ppelberg created this task.Nov 1 2019, 12:58 AM
LGoto triaged this task as High priority.Nov 1 2019, 6:04 PM
LGoto edited projects, added Product-Analytics (Kanban); removed Product-Analytics.

Added context via @MNeisler

Comparison of pre and post edit deployment mobile VE completion rates for both logged in and logged out users.

Editor ExperienceMay 2019September 2019
overall28%14%
over 500 edits61%61%
under 100 edits35%39%
unregistered21%8%

Due to the July 4th drop in edit completion rates for logged-out users, the overall edit completion rate decreased from 28% to 14% for all users (both logged in and logged out).
Let me know if you have any questions or need any changes in prep for Monday's meeting,

One potential explanation for the drop in edit completion rate among unregistered users, that begins in late June-2019 and accelerates on ~4-July-2019, is the start of the A/B test testing the VisualEditor as the default editor on mobile [1].

Reason being: the start of the A/B test would have likely caused a greater number of unregistered, and potentially lower intent users, to open the mobile VisualEditor as a result of it becoming the default editor for 50% of contributors on the 20 wikis participating in the test. [2][3]

Timeline

28-June-2019:

  • Mobile VE is set as the default editor for 50% of contributors to the 20 participating wikis [4]
  • Around this same time, we start to see a decline in edit completion rate among unregistered users [5]

~4-July-2019:

  • Decline in edit completion rate among unregistered contributors accelerates

Next steps

To corroborate the above, @matmarex & @marcella raised the idea of investigating whether the below happened during this same ~28 June – ~15 July time period:

  • The number of unregistered users attempting to make edits using the mobile wikitext editor declined
  • The number of unregistered users attempting to make edits using the mobile VisualEditor increased

  1. T227338
  2. Participating wikis
  3. To edit using mobile VE before the start of the test, unregistered user would have needed to seek mobile VE out on their own by way of switching to it from the mobile wikitext editor.
  4. https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/519667/
  5. See graph:
Decline in edit completion rate among unregistered users begins
MNeisler added a comment.EditedThu, Dec 5, 1:25 AM

I completed a review into the drop in mobile VE edit completion rate among unregistered users and confirmed that the decrease is likely caused by an increase in the number of VE edits attempts by unregistered users following the VE AB test deployment date.

See summary and key observations outlined below.

Investigation Steps:

  • As proposed in T237075#5654915, I reviewed the number of unregistered users attempts using mobile wikitext and Visual Editor to confirm if the drop is due to an increase in unregistered users opening VE on mobile as a result of the VE AB test.
  • Reviewed edit completion rate rates excluding those in the AB test to confirm if the drop is seen only for AB test users.

Summary of key observations.

  • We stopped logging events for mobile users not included in the AB test from June 25th through July 3rd due to the following bug: T221197#5305019. This led to a significant drop in overall edit attempts during these dates. When reviewing edit completion rates, we removed this data as faulty; but I included it in the graphs below so we can more accurately see where the drops and increases around this timeframe are occurring.
  • While we unfortunately can't accurately determine overall mobile edit attempts right around June 28th due to missing data, we can see the number of VE edit attempts by unregistered users doubles compared to previous rates after the VE AB test is turned on (and immediately after we start logging all events again on July 3rd) confirming that there is an increased number of VE edit attempts by unregistered users following the test.
  • After we started logging all events again on July 3rd, the number of wikitext edit attempts by unregistered users did not decrease but returned to similar rates recorded before the June 25th bug. It then sharply increases on July 11th. Based on timing, this increase was not due to the VE as default test but due to the oversampling of mobile wikitext EditAttemptStep events on A/B test wikis on July 11th T227317.

If we remove oversampled users, you can see that the number of attempts by unregistered users on wikitext remains roughly the same.

  • When you remove users included in the AB test, the mobile VE edit completion rates for unregistered users remain fairly stable with the exception of the drop due to the data recording bug around this time frame indicating that the overall decrease in edit completion rate only impacts AB Test users.

Analysis