Page MenuHomePhabricator

VE mobile default: identify wikis for trial
Closed, ResolvedPublic

Description

Task overview

This quarter we will make VE the default mobile editor on at least 3 wikis. This task is about deciding those 3 wikis*.

Note: we will be running an A/B test with these wikis. This means we need to prioritize wikis that have more mobile edit volume/behavior to increase the sample size of each group.

Background

  • We're striving to make editing on mobile web simpler [1]
  • We think contributors will have more success editing on mobile VE given it offers a more structured and visual experience than wikitext. Attributes we think are especially important in a context where time and space can be less abundant than on desktop
  • Difficultly: VE is not the default on mobile; this has meant the overwhelming majority of mobile editors do not interact with it. [2]
  • This makes improving the experience difficult when the majority of mobile contributors are not represented in the quantitative data we've been using to inform our product decisions
  • By making the VE the default editing experience we intend to:
    1. Increase our understanding of the expectations and needs of a broader range of contributors editing on mobile
    2. Learn whether VE, in its current state, is a "better" [3] mobile editing experience than wikitext
    3. Learn how turning on VE as the default mobile editing interface impacts overall edit completion rate

Considerations

  • Enough existing mobile editing behavior (as measured by: mobile edit sessions)
  • Community consent (potential proxy: % of edits made using VE)
  • Language: RTL and LTR represented
  • We have a relationship with someone in this wiki
  • Not going to conflict with another team's work

Definition of "done"

  • Share: with Editing Team
  • Confirm: the set of wikis we are focusing on have no objection to VE as mobile default
  • Publish: wikis we will be trialing mobile VE as default editor with on MediaWiki

Open questions

  • What are the implications of choosing these wikis on our ability to run an A/B test? Time? Precision?
  • How many edit session will we need in our sample size to complete an A/B test in 4 weeks with .95 power (assuming we will not need finer precision than we're currently using for section editing A/B test)
    • Is 3 wikis a necessary constraint considering the scale we'll need to achieve the significance within this time frame?
  • Do the wikis selected meet the criteria we've identified as important?
  • What are the consequences of collaborating with a wiki on this initiative and T221816?

References

  1. Editing Team annual goal: Audiences 2018-19 Annual Plan
  2. Edit sessions recorded in May 2018 | source
EditorEdit sessions% of total
VE20,643~1.0%
Wikitext1,912,178~99%
  1. "Better mobile editing experience": as measured by overall edit completion rate
  2. Mobile edit sessions per wiki
  3. Target-wikis/Q4 sheet.

Event Timeline

ppelberg renamed this task from Identify wikis for trialing VE as default mobile editor to VE mobile default: identify wikis for trial.Apr 17 2019, 7:12 AM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)

This is the spreadsheet @Whatamidoing-WMF and @Neil_P._Quinn_WMF will be using to help identify the wikis we will collaborate with for this quarter's work: Target-wikis/Q4.

7-May meeting notes

Meta
We talked about the selection of wikis as they relate to both VE as default (this task: T221188) and Edit cards (T221816)

Discussed

  • We will work with the same target wikis for edit cards (T221816) and toolbar refresh (T211255)
  • What is the likely reaction of each of the wikis in the Wiki selection sheet to collaborating with us on Edit cards and/or VE as default?
  • We arrived at a preliminary set of wikis for Edit cards and VE as default to work with this quarter (see this sheet for the list)
  • Edit cards At this end of our meeting, we arrived at 3 wikis to collaborate with for mobile edit cards. This amounts to ~12,000 "VE mobile 'readies'/month" (read: 'readies' = ~sessions) [1] ...not a big enough sample size to do an impact analysis that has meaning within a 4-6 week timeframe.
    • There are concerns about collaborating with more than 3 wikis on Edit cards

Next steps

  • What other wikis – beyond those in the Target-wikis/Q4 sheet – can we potentially collaborate with this quarter in order to reach the scale needed to measure the impact – quantitatively – of edit cards ("VE mobile 'readies'/month) and VE as default ("Total mobile edit 'readies'/month") in 4-6 weeks? See: T222803
  • Edit cards: what adjustments are we going to make to our measurement plan considering the current # of wikis we are currently planning to work with (3) will not produce a big enough sample size for us to draw meaningful conclusions from using an impact analysis. <-- @ppelberg to answer this.
    • Variables to affect: number of wikis we work with, size of wikis we work with, length of experiment...

  1. 12,000 "VE mobile 'readies'/month'): see cell G29 in Wiki selection sheet

7-May meeting notes

  • Edit cards: what adjustments are we going to make to our measurement plan considering the current # of wikis we are currently planning to work with (3) will not produce a big enough sample size for us to draw meaningful conclusions from using an impact analysis. <-- @ppelberg to answer this.
    • Variables to affect: number of wikis we work with, size of wikis we work with, length of experiment...

Update

@Whatamidoing-WMF and @Neil_P._Quinn_WMF, I updated Column A (which projects, which wikis) in the Target-wikis/Q4 sheet in an effort to get closer to the scale [in edit sessions] we talked about needing yesterday to run an A/B test for VE as default and an impact analysis for edit cards. This comes as a result of thinking:

  • Maybe we can get the VE as default experiment up and running sooner, giving us a) more time for the experiment to run and b) the flexibility to re-assign some wikis we'd been thinking about working with for VE as default to Edit cards
  • I don't think we can afford to skip quantitatively measuring the impact of our Edit card work via an impact analysis (T222367) considering these results will impact where and how we: a) refine the flow and b) scale this edit card/context item design pattern out to other elements within the editing interface.

Update | 24-May

We have identified a list of candidate wikis to collaborate with this quarter. They are represented in the target wiki sheet.

Ongoing discussion about outreach is happening in: T222803: Evaluate target wiki candidates for Q4 work

The wikis participating in the A/B test are now posted to the /VE mobile default project page.

ppelberg updated the task description. (Show Details)