Page MenuHomePhabricator

Confirm Section Editing instrumentation + A/B test working as we expect
Closed, ResolvedPublicApr 16 2019

Description

Context

On 2-April, Section Editing, and our corresponding A/B test, went live on all remaining wikis.

Open

  • Report: what – if anything – can we draw from ~1 week's of data on Section Editing's impact on edit completion rate?
  • Confirm: our A/B test is working as we expect it to be
  • Confirm: we are gathering the data we will need to report on the "Metrics" listed in the Section Editing Measurement Plan*

Details

Due Date
Apr 16 2019, 1:00 AM

Event Timeline

ppelberg created this task.Apr 8 2019, 11:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 8 2019, 11:13 PM

Hi @ppelberg. Please associate at least one active project with this task (via the Add Action...Change Project Tags dropdown). This will allow others to get notified and see this task when looking at the corresponding project workboard. Thanks a lot! :)

Hi @ppelberg. Please associate at least one active project with this task (via the Add Action...Change Project Tags dropdown). This will allow others to get notified and see this task when looking at the corresponding project workboard. Thanks a lot! :)

Ah, I'd intentionally left the task unassigned so as not to notify anyone while it was still being defined. But I'm now realizing this is not a helpful workflow...noted. Thank you for saying something, @Aklapper.

nshahquinn-wmf set Due Date to Apr 15 2019, 7:00 AM.Apr 10 2019, 10:51 PM
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptApr 10 2019, 10:51 PM
nshahquinn-wmf changed Due Date from Apr 15 2019, 7:00 AM to Apr 16 2019, 1:00 AM.Apr 10 2019, 10:52 PM
nshahquinn-wmf closed this task as Resolved.Apr 17 2019, 1:29 AM

I just checked this out. Since the experiment started, we haven't seen any clear change in the number of mobile visual editor sessions or their completion rate in either the treatment or the control group. The number of mobile VE users in each bucket is very close to even. So, everything looks very sane. Full numbers in this spreadsheet.