Below is the work @Neil_P._Quinn_WMF did to analyze edit completion rate for a previous deliverable; sharing here in case this reference is helpful.

T223339#5274612

Documenting the outcomes of our discussion this morning...

DECIDED

• Considering, EditAttemptStep data only goes as far back as December-2018 our analysis will include data from December-2018 through September-2019
• We will compare edit completion rate in the month before Edit Cards were released to the month after Edit Cards' release as a proxy for understanding their impact, in the context of the Quarter in Review. We will consider the "month before" to be May-2019 and the "month after" to be September-2019 per what we uncovered in T221728#5620554 [1]

...the above is now reflected in the task description.

1. Note: We decided to call the "month after" Edit Cards' release September-2019 (the month following v3's and not v4's release). We decided this because we think having a full month's worth of data to analyze is more valuable than having ~2 weeks worth, which would be necessary if we were to consider v4 to be the "final" Edit Cards release, and because we are assuming the changes introduced v4 will not have a significant impact on the likelihood contributors will complete their edits.

@ppelberg

Here are the initial results.

There was a drop in the overall mobile VE edit completion rate for logged-out users that occurs around July 4th. This led to a similar drop in the overall rate when I included both logged in and logged out users in the data. I need to investigate this further but for the sake of time, I'm sharing current results now. For comparison, I included both (1) a summary of trends looking only at logged in users and (2) a summary of trends including both logged in and logged out users.

A couple of notes re timeline:

• We started recording data but did not have clean, complete data until February 8, 2019. I looked at trends starting from that date until September 30, 2019. I'll update the description to reflect this change.
• We did not start oversampling the mobile visual editor events until March 6th (T212253), which doesn't impact the overall trends but it is likely causing some of the larger fluctuations you see before then in the time series charts below.
• For the editor experience levels: I used under 100 cumulative edits and over 500 cumulative edits to be consistent with our current definitions of junior and senior contributors but let me know if there are other experience breakouts that would be useful.

(1) Mobile VE edit completion rates for only registered users

*Overall Trend

There is a steady increase in mobile VE edit completion rates for all registered users.

• Pre and Post Edit Deployment

Comparison of pre and post edit deployment mobile ve completion rates for registered users.

Comparing the month before edit card release (May 2019) to after (September 2019), there was about a 10% overall increase in the edit completion rate for registered users. The rate of edit completion for senior contributors (over 500 cumulative edits) remained about the same but there was an 11% increase for junior contributors.

(2) Mobile VE edit completion rates including logged out users

• Overall Trend

When you include logged out users, there is a drop in the edit completion rate for unregistered users that occurs around July 4, 2019. This leads to a similar decrease in the overall edit completion rate.

• Pre and Post Edit Deployment

Comparison of pre and post edit deployment mobile VE completion rates for both logged in and logged out users.

Due to the July 4th drop in edit completion rates for logged-out users, the overall edit completion rate decreased from 28% to 14% for all users (both logged in and logged out).

Let me know if you have any questions or need any changes in prep for Monday's meeting,

This is looking great, @MNeisler. Nice. A few comments and questions below...

Comments

• The drop ~4-July drop in the edit completion rate among logged-out users is curious – I'm glad you highlighted this and that we will explore this further. I've created a new task for this investigation. Tho, if you think it is better suited as a part of this task, please boldly close this one as a duplicate: T237075

Questions

• 1. How have we defined "edit completion rate" in this context? I'm just wanting to be sure we're being consistent with how we've defined it previously [1] so we can make 1:1 comparisons.
• 2. Are we able to bucket contributors using the edit count buckets that are available in Superset? [2] I appreciate there is value in keeping this bucketing consistent with how we are segmenting contributors as part of the OWC2020 work. Although in this context, I think it's worth being able to have a more granular understanding of edit completion rate and to use use buckets people in the audience will be familiar with.
• 3. The pre- and post-Edit Cards improvement to edit completion rate among registered users (overall and those with <100 edits) is encouraging...what – if any – information do you think we still need before you'd feel comfortable saying something like, "Edit Cards likely caused a ~10% overall increase in the edit completion rate for registered users."?
  • A thought that came to mind: How many edit sessions involve contributors interacting with Edit Cards?
  • I appreciate we will not have time to get to the bottom of this before Monday. In which case, please feel no obligation to engage with this question before then. Sharing here so we don't lose track of it. I suspect answering this particular question as part of T221728 makes more sense.

1. T223339#5274612
2. Edit count buckets: 1-4; 5-99; 100-999; 1,000+; unregistered ---

Edited Fri., 1-Nov @ 7:46am PST: //added questions "3."

I added question "3." to T236860#5625456. Documenting here so the edit does not go unnoticed.

Comments

• The drop ~4-July drop in the edit completion rate among logged-out users is curious – I'm glad you highlighted this and that we will explore this further. I've created a new task for this investigation. Tho, if you think it is better suited as a part of this task, please boldly close this one as a duplicate: T237075

That's perfect. Thanks!

Questions

• 1. How have we defined "edit completion rate" in this context? I'm just wanting to be sure we're being consistent with how we've defined it previously [1] so we can make 1:1 comparisons.

Edit completion rate is defined the same way it was in the previous reports: "the proportion of attempts (anything that reaches "ready) that reach save success".

• 2. Are we able to bucket contributors using the edit count buckets that are available in Superset? [2] I appreciate there is value in keeping this bucketing consistent with how we are segmenting contributors as part of the OWC2020 work. Although in this context, I think it's worth being able to have a more granular understanding of edit completion rate and to use use buckets people in the audience will be familiar with.

Yes. That makes sense and it's a pretty easy change. I'll redo the breakdowns this afternoon.

• 3. The pre- and post-Edit Cards improvement to edit completion rate among registered users (overall and those with <100 edits) is encouraging...what – if any – information do you think we still need before you'd feel comfortable saying something like, "Edit Cards likely caused a ~10% overall increase in the edit completion rate for registered users."?

Yeah, I'll need a little more time to think through this one. Initial thoughts - Based on the data I have right now, I'd be comfortable saying that there was a "[X]% increase in edit completion rates for registered users before and after deployment of edit cards". There are a number of other variables such as seasonality trends and impacts from any other changes to the mobile visual editor done around this timeframe that might also lead to a portion of that increase. The suggestion you offered below would help isolate the impact. I can look into what's possible based on given instrumentation as part of the full edit card analysis.

• A thought that came to mind: How many edit sessions involve contributors interacting with Edit Cards?
• I appreciate we will not have time to get to the bottom of this before Monday. In which case, please feel no obligation to engage with this question before then. Sharing here so we don't lose track of it. I suspect answering this particular question as part of T221728 makes more sense.

Here is an updated summary of results with edit count breakdown revised to the following edit count buckets: 1-4; 5-99; 100-999; 1,000+; unregistered. See notebook for additional details and charts.

(1) Mobile VE edit completion rates for only registered users

*Overall Trend

• Pre and Post Edit Deployment

Comparison of pre and post edit deployment mobile ve completion rates for registered users.

(2) Mobile VE edit completion rates including logged out users

• Overall Trend

• Pre and Post Edit Deployment

Comparison of pre and post edit deployment mobile VE completion rates for both logged in and logged out users.