Page MenuHomePhabricator

Help panel: instrumentation
Closed, ResolvedPublic

Description

This task will start with @nettrom_WMF and will probably have sub-tasks.

We will want to have insight into how users interact with the feature. Some of the things we will want to know are:

  • What was the user doing (or attempting to do) right before interacting with the feature?
  • How often do users open the feature?
  • How often do users dismiss the feature?
  • If there are multiple options inside the feature, how often are those used?

We will also want to connect this with the data we are gathering via "Understanding first day" so that we can see the impact that this feature has on the broader experience for the new editor.

Event Timeline

MMiller_WMF updated the task description. (Show Details)Oct 10 2018, 11:43 PM
mpopov moved this task from Triage to Backlog on the Product-Analytics board.Oct 26 2018, 6:34 PM
mpopov triaged this task as Normal priority.Oct 26 2018, 6:37 PM
MMiller_WMF renamed this task from Help desk: instrumentation to Help pane: instrumentation.Nov 13 2018, 11:31 PM
MMiller_WMF updated the task description. (Show Details)Nov 19 2018, 10:27 PM

Product and Analytics met to develop the following specifications:
Summary

Higher level

  • Does the presence of the help pane increase editor activation and retention?
  • Possible caveat: if the help pane makes it easier for a user to make their first edit, they are arguably being activated, thus we’re altering the probability of being activated. As a result, we might want to modify activation to only concern edits in the Main namespace.
  • Does the presence of the help pane lead to questions being asked and answered?
  • Does the presence of the help pane increase edit activity?
  • Perhaps this question is also better phrased in terms of “constructive edits”.

Lower level

  • What are users doing (or attempting to do) right before interacting with the help pane?
  • What percent of users open the help pane?
  • What percent close the help pane without using any part of it?
  • What percent click a help link, and which link?
  • How far do users go through the process of asking a question?
  • Type in the box
  • Type in the box -> click on the button
  • Type in the box -> click on the button -> submit
  • Type in the box -> click on the button -> submit -> click a link in confirmation
  • How often do users use the help pane more than once?
  • “Use”: clicks a link or asks a question.
  • Does the presence of the help pane lead to a lower probability of abandoning an edit session? (time to help) (secondary: it might increase time to abandonment)
  • What percent of questions get answered?
  • How quickly do questions get answered?
  • What percent of users return to view the answer to their question? (extend EditorJourney beyond 24 hours for help desk? Or log only if someone has asked a question on the help desk)?
  • What kind of questions to people ask?
  • What kind of questions do and do not get answered?
  • What percent of users had already read some type of help content before using the help pane?
  • Do something before clicking
  • Click on CTA
  • Click on one of the help links
  • Type some text in the question box
  • Click “Ask a question”
  • Submit
  • Click on a link in the confirmation dialogue
  • Close / collapse
  • Do something afterwards.

What was the user doing (or attempting to do) right before interacting with the feature?
How often do users open the feature?
How often do users dismiss the feature?
If there are multiple options inside the feature, how often are those used?

Product and Analytics met to develop the following specifications:
Possible caveat: if the help pane makes it easier for a user to make their first edit, they are arguably being activated, thus we’re altering the probability of being activated. As a result, we might want to modify activation to only concern edits in the Main namespace.

Good catch. I think the pane would also be useful on Categories (to edit them) or on Help pages (juste because they may have a question about how to help others). For communities having those, on Portals.

  • Does the presence of the help pane lead to questions being asked and answered?

It would certainly be possible if each question was a separate page, not a section potentially archived.

All questions about measuring the interaction on the questions are very interesting.

  • What kind of questions do and do not get answered?

That one would be nice to have, so that we can see what blocks experienced users (and if we can help)

MMiller_WMF renamed this task from Help pane: instrumentation to Help panel: instrumentation.Nov 29 2018, 7:14 PM
nettrom_WMF moved this task from Backlog to Doing on the Product-Analytics board.Dec 5 2018, 10:50 PM

The questions that were listed in this ticket have grown to become the full "Help Panel -- measurement specifications" document: https://docs.google.com/document/d/1rnk-eT092c3CHESBgHxwN0udiXRP8lqgbENWQ-1RhBE/edit#

MMiller_WMF added a subscriber: kzimmerman.

We have approval from the Manager of Product Analytics, as required by the Instrumentation DACI.

@MMiller_WMF @JTannerWMF I've reviewed with @nettrom_WMF and approved Research Questions (DACI B), Instrumentation (DACI C), and Schema Creation (E). Data Dictionary (H) doesn't yet exist.

I think that's everything - let me know if you need anything else!

nettrom_WMF closed this task as Resolved.Feb 21 2019, 8:20 PM

The instrumentation document has gone through the DACI and received approval. We've put the instrumentation in production and monitored the data periodically, and things seem to be going well. Am therefore closing this task.