Page MenuHomePhabricator

Check on Section Editing A/B test results
Closed, ResolvedPublic

Description

Overview

We should now have enough data to begin analyzing the impact of the Section Editing A/B test (see: T218851). This task is about doing a quick check to see what impacts the test has had.

"Done"

Answer the following questions:

  • Has there been a clear change in the number of mobile VE edit sessions?
  • Has there been a clear change in the edit completion rate in either the treatment or control group?
  • What – if anything - should we be concerned about?

Generate graphs showing:

  • Edit completion rate: treatment vs. control
  • Edit sessions: treatment vs. control
  • Users: treatment vs. control

References

Event Timeline

ppelberg created this task.May 14 2019, 4:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2019, 4:58 PM
ppelberg updated the task description. (Show Details)May 16 2019, 1:39 AM
ppelberg updated the task description. (Show Details)
Neil_P._Quinn_WMF moved this task from Icebox to Next Up on the Product-Analytics board.

Mikhail is going to take care of this. 👏👏👏

mpopov added a comment.EditedJun 14 2019, 3:16 PM

Okay, here are the daily user counts, VE session counts, sessions/user rates, and edit completion (success) rates:

Key takeaways from the analysis:

  • No clear change in sessions/user
    • The average number of sessions per user was 1.94 in the control group and 1.948 in the treatment group; this change was not statistically significant
    • We're good to keep section editing on without worrying that it may negatively impact this metric
  • Clear improvement in completion rate
    • There was statistically significant (p-value = 0.034) improvement of 0.98% (in probability of completion) on average
    • This roughly translates to 1 more completed VE edit for every 100 started if section editing is enabled than if it's not; or, viewed in another way, 1 fewer abandoned VE edit for every 100 started

@ppelberg: My analysis of the data suggests that we should do a full rollout of section editing on mobile VE :D

mpopov moved this task from Next Up to Doing on the Product-Analytics board.Jun 14 2019, 3:24 PM

My analysis of the data suggests that we should do a full rollout of section editing on mobile VE :D

This is great news. Thanks for the analysis @mpopov

My analysis of the data suggests that we should do a full rollout of section editing on mobile VE :D

This is great news. Thanks for the analysis @mpopov

+1 – thank you for stepping in and stepping up to bring this all together, @mpopov! We will be turning off the A/B test and rolling out Section Editing to all mobile VE today. See: T225645#5259886

Thanks @mpopov and @Neil_P._Quinn_WMF! I'm closing this as resolved.

kzimmerman closed this task as Resolved.Jun 18 2019, 2:54 PM