Page MenuHomePhabricator

Instrument structured workflows and their non-stuctured counterparts in the mobile visual editor
Closed, ResolvedPublic

Description

This task [and its subtasks] is about the work involved with instrumenting both the existing and revised [1] adding/modifying links and citations workflows on mobile, in VE.

Open questions

  • How will we be able to interact with the data our instrumentation/event logging is gathering?
  • Will we need to oversample Edit events?
  • Confirm: As part of this work, we will not be creating any new schema. This question is relevant to the analytics DACI steps (below).

Background

We know what proportion of edits using the visual editor involve using common editing features. [2] We do not yet know how many contributors take which actions within these workflows. We are pursuing instrumentation to help us answer questions related to this second part. Questions like...

  • What % of contributors who start to add/modify a link and citation are successful in doing so? How do these proportions compare across the existing and revised workflows?
    • This kind of information could be helpful in identifying opportunities to make adjustments that will help these contributors find their way (read: be successful in) adding/modifying links/citations.
  • Do "Edit cards" encourage more people to engage more deeply with adding/modifying links and citations?
    • Asked another way: "What percentage of contributors who are shown the edit card (revised) or context item (existing) end up engaging with it?"
  • What % of edit sessions, in which contributors successfully add/modify a link, result in them attempting to save their edits?" How do the revised and end existing flows compare in this way?
  • Do "Edit cards" cause more people to add/modify links and citations more efficiently?

This work sits within our larger effort to improve the mobile edit completion rate. Work that depends on us being able to know what contributors actually do after starting an editing session.

"Done"

StatusTaskOwners
Define research questions instrumentation needs to answer@ppelberg
Define metrics to be derived from instrumentation@ppelberg @Neil_P._Quinn_WMF
Define data instrumentation will need to generate@ppelberg @Neil_P._Quinn_WMF
Confirm instrumentation definition@DLynch @Neil_P._Quinn_WMF
Define schema and queries and other attributes of the logging@DLynch @Neil_P._Quinn_WMF
Create queries that will generate results@Neil_P._Quinn_WMF
Make legal aware that schema privacy review will be needed@JTannerWMF
Email "privacy@" with link to Phabricator ticket for review@JTannerWMF
Finalize schema@DLynch @Neil_P._Quinn_WMF
Implement instrumentation@DLynch
Test instrumentation@DLynch @Neil_P._Quinn_WMF @ppelberg
Update data dictionary@Neil_P._Quinn_WMF
Activate production level logging@DLynch

Instrumentation needs

The instrumentation specification is defined in this doc (publicly commentable).

Resources


References

  1. Revised workflows: https://phabricator.wikimedia.org/T221247
  2. Feature use data: https://www.mediawiki.org/wiki/VisualEditor_on_mobile_report

Related Objects

Event Timeline

ppelberg created this task.May 2 2019, 2:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 2 2019, 2:44 PM
ppelberg updated the task description. (Show Details)May 2 2019, 4:08 PM
ppelberg updated the task description. (Show Details)May 2 2019, 5:00 PM
ppelberg updated the task description. (Show Details)May 2 2019, 5:58 PM
ppelberg updated the task description. (Show Details)May 2 2019, 6:23 PM
ppelberg updated the task description. (Show Details)May 2 2019, 7:52 PM
ppelberg added a subscriber: nshahquinn-wmf.
ppelberg added a subscriber: DLynch.EditedMay 2 2019, 8:34 PM

Today, @DLynch, @Neil_P._Quinn_WMF and I met to talk about instrumenting the revised and existing adding/modifying linking and citation flows.

Discussed:

  • It's difficult to make an exact 1:1 comparison between the revised and existing flows because the interactions that trigger the edit card (revised) and context bar (existing) being shown will be different.
  • In order to create an exhaustive list of the events/actions we will log, we first need to be able to review the completed wireframes
  • We shouldn't need to make substantial changes to Schema:VisualEditorFeatureUse. Although, we won't be certain until we come up with an exhaustive list of the events we need to log. See T222395 & T222396
  • It would be difficult to track events related to editing link labels within the document; it's hard to know when someone has started and stopped editing a link label
  • It may be difficult to accurately calculate the time between events in each workflow without making revisions to Schema:Edit timing

Next steps

  • T222395 Define event logging needs within the *existing* adding/modifying links workflow
  • T221296 Instrument existing "adding/modifying links" flow

(Revised adding/modifying links wireframes are completed: T220385)

  • T222396 Define event logging needs within the *revised* adding/modifying links workflow
  • T221249 Instrument revised "adding/modifying links" flow
ppelberg updated the task description. (Show Details)May 2 2019, 8:36 PM
nshahquinn-wmf renamed this task from Instrumentation to Instrument structured workflows and their non-stuctured counterparts in the mobile visual editor.May 3 2019, 2:20 AM
ppelberg updated the task description. (Show Details)May 9 2019, 1:39 AM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)May 9 2019, 7:11 PM

I've put in the instrumentation sketch for the existing link workflows (comments very welcome!).

I will do the sketch for the existing citation workflows tomorrow.

ppelberg updated the task description. (Show Details)May 16 2019, 1:34 AM
nshahquinn-wmf removed nshahquinn-wmf as the assignee of this task.May 20 2019, 10:34 PM
nshahquinn-wmf moved this task from Doing to Tracking on the Product-Analytics board.
ppelberg updated the task description. (Show Details)May 23 2019, 12:40 AM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg added a subscriber: JTannerWMF.
ppelberg updated the task description. (Show Details)May 23 2019, 8:27 PM
ppelberg closed this task as Resolved.Sep 3 2019, 1:25 AM
ppelberg claimed this task.