Page MenuHomePhabricator

Determine decision points for VE as default
Closed, ResolvedPublic

Description

Task overview

This task involves the work with creating a plan for how we will decide to set VE as the default mobile editing interface.

Context

This plan will include a table with two columns: Scenario and Plan of action.

The "Scenario" column should include a measure about an activity (i.e. edit completion rate).

The "Plan of action" column should include the set of steps we plan to take should we encounter a given scenario.

References

Decision points are being documented here: Measurement Plan/VE default

"Done"

  • Create a "decision point" table that includes two columns: Scenario and Plan of action

Event Timeline

ppelberg created this task.Jun 27 2019, 4:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2019, 4:59 AM
ppelberg updated the task description. (Show Details)Jun 27 2019, 3:47 PM
ppelberg renamed this task from Determine leading indicators for VE default A/B test to Determine decision points for VE as default.Jun 28 2019, 7:38 PM
ppelberg updated the task description. (Show Details)
ScenarioDescriptionAction(s)Notes
Scenario 1 default-visual has a "higher" edit completion rate than default-sourceTurn on [1] VE as mobile default on all wikis for: A) All contributors who do not have a preference set and B) All contributors who have fewer than 100 edits on that wiki
Scenario 2default-visual has a "lower" edit completion rate than default-sourceFirst: leave wikitext as default on all non-test wikis; Second: run additional analysis (qualitative and quantitative) to understand what might be the cause for VE having a lower edit completion rate; explore design interventions (e.g. onboarding) that make it more clear to "newer" contributors VE is available
Scenario 3default-visual and default-source have "the same" edit completion ratesFirst: compare A/B test groups across other metrics (e.g. # of completed edits, revert rate, number of attempts); Next: run a qualitative test to determine which interface newcomers have more success with; assuming VE doesn't show to encourage lower quality edits and outcomes of qual. tests are consistent with what prior tests and research have shown (VE better for newcomers), turn on [1] VE as mobile default on all wikis (see "Scenario 1")

Definitions: "higher" / "lower" / "the same"

  • Definitions around edit completion rate (ECR) being "higher" / "lower" / "the same" should fall on analysis to determine what qualifies as a “statistically significant” result.
  • We are going to let our statistical tests determine whether a change in the average user’s ECR is large enough to be considered significantly higher or lower. (Specifically, we plan to use a [Mann-Whitney U test](Mann-Whitney U test) or, if time allows, a multilevel model incorporating per-user and per-wiki fixed effects).

  1. Turn on VE as default for all wikis: before doing this, we would need to make communities aware