Page MenuHomePhabricator

Check on Section Editing A/B test results
Closed, ResolvedPublic

Description

Overview

We should now have enough data to begin analyzing the impact of the Section Editing A/B test (see: T218851). This task is about doing a quick check to see what impacts the test has had.

"Done"

Answer the following questions:

  • Has there been a clear change in the number of mobile VE edit sessions?
  • Has there been a clear change in the edit completion rate in either the treatment or control group?
  • What – if anything - should we be concerned about?

Generate graphs showing:

  • Edit completion rate: treatment vs. control
  • Edit sessions: treatment vs. control
  • Users: treatment vs. control

References

Event Timeline

Mikhail is going to take care of this. 👏👏👏

Okay, here are the daily user counts, VE session counts, sessions/user rates, and edit completion (success) rates:

section-editing-daily.png (1×1 px, 232 KB)

Key takeaways from the analysis:

  • No clear change in sessions/user
    • The average number of sessions per user was 1.94 in the control group and 1.948 in the treatment group; this change was not statistically significant
    • We're good to keep section editing on without worrying that it may negatively impact this metric
  • Clear improvement in completion rate
    • There was statistically significant (p-value = 0.034) improvement of 0.98% (in probability of completion) on average
    • This roughly translates to 1 more completed VE edit for every 100 started if section editing is enabled than if it's not; or, viewed in another way, 1 fewer abandoned VE edit for every 100 started

@ppelberg: My analysis of the data suggests that we should do a full rollout of section editing on mobile VE :D

My analysis of the data suggests that we should do a full rollout of section editing on mobile VE :D

This is great news. Thanks for the analysis @mpopov

My analysis of the data suggests that we should do a full rollout of section editing on mobile VE :D

This is great news. Thanks for the analysis @mpopov

+1 – thank you for stepping in and stepping up to bring this all together, @mpopov! We will be turning off the A/B test and rolling out Section Editing to all mobile VE today. See: T225645#5259886

Thanks @mpopov and @Neil_P._Quinn_WMF! I'm closing this as resolved.