Page MenuHomePhabricator

VE mobile default: identify wikis for trial
Closed, ResolvedPublic

Description

Task overview

This quarter we will make VE the default mobile editor on at least 3 wikis. This task is about deciding those 3 wikis*.

Note: we will be running an A/B test with these wikis. This means we need to prioritize wikis that have more mobile edit volume/behavior to increase the sample size of each group.

Background

  • We're striving to make editing on mobile web simpler [1]
  • We think contributors will have more success editing on mobile VE given it offers a more structured and visual experience than wikitext. Attributes we think are especially important in a context where time and space can be less abundant than on desktop
  • Difficultly: VE is not the default on mobile; this has meant the overwhelming majority of mobile editors do not interact with it. [2]
  • This makes improving the experience difficult when the majority of mobile contributors are not represented in the quantitative data we've been using to inform our product decisions
  • By making the VE the default editing experience we intend to:
    1. Increase our understanding of the expectations and needs of a broader range of contributors editing on mobile
    2. Learn whether VE, in its current state, is a "better" [3] mobile editing experience than wikitext
    3. Learn how turning on VE as the default mobile editing interface impacts overall edit completion rate

Considerations

  • Enough existing mobile editing behavior (as measured by: mobile edit sessions)
  • Community consent (potential proxy: % of edits made using VE)
  • Language: RTL and LTR represented
  • We have a relationship with someone in this wiki
  • Not going to conflict with another team's work

Definition of "done"

  • Share: with Editing Team
  • Confirm: the set of wikis we are focusing on have no objection to VE as mobile default
  • Publish: wikis we will be trialing mobile VE as default editor with on MediaWiki

Open questions

  • What are the implications of choosing these wikis on our ability to run an A/B test? Time? Precision?
  • How many edit session will we need in our sample size to complete an A/B test in 4 weeks with .95 power (assuming we will not need finer precision than we're currently using for section editing A/B test)
    • Is 3 wikis a necessary constraint considering the scale we'll need to achieve the significance within this time frame?
  • Do the wikis selected meet the criteria we've identified as important?
  • What are the consequences of collaborating with a wiki on this initiative and T221816?

References

  1. Editing Team annual goal: Audiences 2018-19 Annual Plan
  2. Edit sessions recorded in May 2018 | source
EditorEdit sessions% of total
VE20,643~1.0%
Wikitext1,912,178~99%
  1. "Better mobile editing experience": as measured by overall edit completion rate
  2. Mobile edit sessions per wiki
  3. Target-wikis/Q4 sheet.

Event Timeline

ppelberg created this task.Apr 17 2019, 6:13 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 17 2019, 6:13 AM
ppelberg renamed this task from Identify wikis for trialing VE as default mobile editor to VE mobile default: identify wikis for trial.Apr 17 2019, 7:12 AM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)Apr 25 2019, 3:33 PM
ppelberg added a subscriber: Neil_P._Quinn_WMF.
ppelberg updated the task description. (Show Details)Apr 25 2019, 3:45 PM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)Apr 25 2019, 4:47 PM
ppelberg updated the task description. (Show Details)Apr 25 2019, 6:29 PM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)Apr 25 2019, 6:34 PM
ppelberg updated the task description. (Show Details)May 6 2019, 10:35 PM
ppelberg updated the task description. (Show Details)May 7 2019, 12:19 AM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)May 7 2019, 12:38 AM
ppelberg updated the task description. (Show Details)May 7 2019, 1:47 AM
ppelberg updated the task description. (Show Details)May 7 2019, 4:33 PM

This is the spreadsheet @Whatamidoing-WMF and @Neil_P._Quinn_WMF will be using to help identify the wikis we will collaborate with for this quarter's work: Target-wikis/Q4.

ppelberg updated the task description. (Show Details)May 8 2019, 1:32 AM
ppelberg added a comment.EditedMay 8 2019, 3:18 PM

7-May meeting notes

Meta
We talked about the selection of wikis as they relate to both VE as default (this task: T221188) and Edit cards (T221816)

Discussed

  • We will work with the same target wikis for edit cards (T221816) and toolbar refresh (T211255)
  • What is the likely reaction of each of the wikis in the Wiki selection sheet to collaborating with us on Edit cards and/or VE as default?
  • We arrived at a preliminary set of wikis for Edit cards and VE as default to work with this quarter (see this sheet for the list)
  • Edit cards At this end of our meeting, we arrived at 3 wikis to collaborate with for mobile edit cards. This amounts to ~12,000 "VE mobile 'readies'/month" (read: 'readies' = ~sessions) [1] ...not a big enough sample size to do an impact analysis that has meaning within a 4-6 week timeframe.
    • There are concerns about collaborating with more than 3 wikis on Edit cards

Next steps

  • What other wikis – beyond those in the Target-wikis/Q4 sheet – can we potentially collaborate with this quarter in order to reach the scale needed to measure the impact – quantitatively – of edit cards ("VE mobile 'readies'/month) and VE as default ("Total mobile edit 'readies'/month") in 4-6 weeks? See: T222803
  • Edit cards: what adjustments are we going to make to our measurement plan considering the current # of wikis we are currently planning to work with (3) will not produce a big enough sample size for us to draw meaningful conclusions from using an impact analysis. <-- @ppelberg to answer this.
    • Variables to affect: number of wikis we work with, size of wikis we work with, length of experiment...

  1. 12,000 "VE mobile 'readies'/month'): see cell G29 in Wiki selection sheet

7-May meeting notes

  • Edit cards: what adjustments are we going to make to our measurement plan considering the current # of wikis we are currently planning to work with (3) will not produce a big enough sample size for us to draw meaningful conclusions from using an impact analysis. <-- @ppelberg to answer this.
    • Variables to affect: number of wikis we work with, size of wikis we work with, length of experiment...

Update

@Whatamidoing-WMF and @Neil_P._Quinn_WMF, I updated Column A (which projects, which wikis) in the Target-wikis/Q4 sheet in an effort to get closer to the scale [in edit sessions] we talked about needing yesterday to run an A/B test for VE as default and an impact analysis for edit cards. This comes as a result of thinking:

  • Maybe we can get the VE as default experiment up and running sooner, giving us a) more time for the experiment to run and b) the flexibility to re-assign some wikis we'd been thinking about working with for VE as default to Edit cards
  • I don't think we can afford to skip quantitatively measuring the impact of our Edit card work via an impact analysis (T222367) considering these results will impact where and how we: a) refine the flow and b) scale this edit card/context item design pattern out to other elements within the editing interface.

Update | 24-May

We have identified a list of candidate wikis to collaborate with this quarter. They are represented in the target wiki sheet.

Ongoing discussion about outreach is happening in: T222803: Evaluate target wiki candidates for Q4 work

ppelberg updated the task description. (Show Details)Jun 21 2019, 2:19 PM

The wikis participating in the A/B test are now posted to the /VE mobile default project page.

ppelberg closed this task as Resolved.Jun 28 2019, 12:05 AM
ppelberg updated the task description. (Show Details)