Added context via @MNeisler

In T236860#5624424, @MNeisler wrote:

Comparison of pre and post edit deployment mobile VE completion rates for both logged in and logged out users.

Editor Experience	May 2019	September 2019
overall	28%	14%
over 500 edits	61%	61%
under 100 edits	35%	39%
unregistered	21%	8%

Due to the July 4th drop in edit completion rates for logged-out users, the overall edit completion rate decreased from 28% to 14% for all users (both logged in and logged out).

Let me know if you have any questions or need any changes in prep for Monday's meeting,

One potential explanation for the drop in edit completion rate among unregistered users, that begins in late June-2019 and accelerates on ~4-July-2019, is the start of the A/B test testing the VisualEditor as the default editor on mobile [1].

Reason being: the start of the A/B test would have likely caused a greater number of unregistered, and potentially lower intent users, to open the mobile VisualEditor as a result of it becoming the default editor for 50% of contributors on the 20 wikis participating in the test. [2][3]

Timeline

28-June-2019:

• Mobile VE is set as the default editor for 50% of contributors to the 20 participating wikis [4]
• Around this same time, we start to see a decline in edit completion rate among unregistered users [5]

~4-July-2019:

• Decline in edit completion rate among unregistered contributors accelerates

Next steps

To corroborate the above, @matmarex & @marcella raised the idea of investigating whether the below happened during this same ~28 June – ~15 July time period:

• The number of unregistered users attempting to make edits using the mobile wikitext editor declined
• The number of unregistered users attempting to make edits using the mobile VisualEditor increased

1. T227338
2. Participating wikis
3. To edit using mobile VE before the start of the test, unregistered user would have needed to seek mobile VE out on their own by way of switching to it from the mobile wikitext editor.
4. https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/519667/
5. See graph:

Decline in edit completion rate among unregistered users begins

I completed a review into the drop in mobile VE edit completion rate among unregistered users and confirmed that the decrease is likely caused by an increase in the number of VE edits attempts by unregistered users following the VE AB test deployment date.

See summary and key observations outlined below.

Investigation Steps:

• As proposed in T237075#5654915, I reviewed the number of unregistered users attempts using mobile wikitext and Visual Editor to confirm if the drop is due to an increase in unregistered users opening VE on mobile as a result of the VE AB test.
• Reviewed edit completion rate rates excluding those in the AB test to confirm if the drop is seen only for AB test users.

Summary of key observations.

• We stopped logging events for mobile users not included in the AB test from June 25th through July 3rd due to the following bug: T221197#5305019. This led to a significant drop in overall edit attempts during these dates. When reviewing edit completion rates, we removed this data as faulty; but I included it in the graphs below so we can more accurately see where the drops and increases around this timeframe are occurring.
• While we unfortunately can't accurately determine overall mobile edit attempts right around June 28th due to missing data, we can see the number of VE edit attempts by unregistered users doubles compared to previous rates after the VE AB test is turned on (and immediately after we start logging all events again on July 3rd) confirming that there is an increased number of VE edit attempts by unregistered users following the test.
• After we started logging all events again on July 3rd, the number of wikitext edit attempts by unregistered users did not decrease but returned to similar rates recorded before the June 25th bug. It then sharply increases on July 11th. Based on timing, this increase was not due to the VE as default test but due to the oversampling of mobile wikitext EditAttemptStep events on A/B test wikis on July 11th T227317.

If we remove oversampled users, you can see that the number of attempts by unregistered users on wikitext remains roughly the same.

• When you remove users included in the AB test, the mobile VE edit completion rates for unregistered users remain fairly stable with the exception of the drop due to the data recording bug around this time frame indicating that the overall decrease in edit completion rate only impacts AB Test users.

Analysis

This looks great, @MNeisler – thank you for the clear explanation and for getting to the bottom of this [as best as we can tell].