Determine decision points for VE as default
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	Jun 27 2019, 4:59 AM

Description

Task overview

This task involves the work with creating a plan for how we will decide to set VE as the default mobile editing interface.

Context

This plan will include a table with two columns: Scenario and Plan of action.

The "Scenario" column should include a measure about an activity (i.e. edit completion rate).

The "Plan of action" column should include the set of steps we plan to take should we encounter a given scenario.

References

Decision points are being documented here: Measurement Plan/VE default

"Done"

Create a "decision point" table that includes two columns: Scenario and Plan of action

Related Objects
Search...

Status	Assigned	Task
Open	None	T255327 [Epic] Evaluate which editing interface should be shown by default
Open	None	T227338 Test visual editor as the default mobile editor on select wikis
Resolved	ppelberg	T226687 Determine decision points for VE as default

Event Timeline

ppelberg created this task.Jun 27 2019, 4:59 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2019, 4:59 AM

ppelberg added a parent task: T221185: Prepare visual editor to be a default mobile editor.Jun 27 2019, 5:00 AM

ppelberg updated the task description. (Show Details)Jun 27 2019, 3:47 PM

ppelberg renamed this task from Determine leading indicators for VE default A/B test to Determine decision points for VE as default.Jun 28 2019, 7:38 PM

ppelberg updated the task description. (Show Details)

nshahquinn-wmf removed a parent task: T221185: Prepare visual editor to be a default mobile editor.Jul 5 2019, 3:50 PM

nshahquinn-wmf added a parent task: T227338: Test visual editor as the default mobile editor on select wikis.

JTannerWMF moved this task from Incoming to Ready for Development on the VisualEditor (Current work) board.Aug 21 2019, 11:54 AM

Scenario	Description	Action(s)	Notes
Scenario 1	`default-visual` has a "higher" edit completion rate than `default-source`	Turn on [1] VE as mobile default on all wikis for: A) All contributors who do not have a preference set and B) All contributors who have fewer than 100 edits on that wiki
Scenario 2	`default-visual` has a "lower" edit completion rate than `default-source`	First: leave wikitext as default on all non-test wikis; Second: run additional analysis (qualitative and quantitative) to understand what might be the cause for VE having a lower edit completion rate; explore design interventions (e.g. onboarding) that make it more clear to "newer" contributors VE is available
Scenario 3	`default-visual` and `default-source` have "the same" edit completion rates	First: compare A/B test groups across other metrics (e.g. # of completed edits, revert rate, number of attempts); Next: run a qualitative test to determine which interface newcomers have more success with; assuming VE doesn't show to encourage lower quality edits and outcomes of qual. tests are consistent with what prior tests and research have shown (VE better for newcomers), turn on [1] VE as mobile default on all wikis (see "Scenario 1")

Definitions: "higher" / "lower" / "the same"

Definitions around edit completion rate (ECR) being "higher" / "lower" / "the same" should fall on analysis to determine what qualifies as a “statistically significant” result.
We are going to let our statistical tests determine whether a change in the average user’s ECR is large enough to be considered significantly higher or lower. (Specifically, we plan to use a [Mann-Whitney U test](Mann-Whitney U test) or, if time allows, a multilevel model incorporating per-user and per-wiki fixed effects).

Turn on VE as default for all wikis: before doing this, we would need to make communities aware

These decision points have been posted on the project page: https://www.mediawiki.org/wiki/VisualEditor_on_mobile/VE_mobile_default#18_September_2019_–_Test_analysis

ppelberg mentioned this in T221198: VE mobile default: analyze A/B test results.Jun 6 2020, 1:27 AM

Determine decision points for VE as defaultClosed, ResolvedPublicActions