Product and Analytics met to develop the following specifications:
Summary

Higher level

• Does the presence of the help pane increase editor activation and retention?
• Possible caveat: if the help pane makes it easier for a user to make their first edit, they are arguably being activated, thus we’re altering the probability of being activated. As a result, we might want to modify activation to only concern edits in the Main namespace.
• Does the presence of the help pane lead to questions being asked and answered?
• Does the presence of the help pane increase edit activity?
• Perhaps this question is also better phrased in terms of “constructive edits”.

Lower level

• What are users doing (or attempting to do) right before interacting with the help pane?
• What percent of users open the help pane?
• What percent close the help pane without using any part of it?
• What percent click a help link, and which link?
• How far do users go through the process of asking a question?
• Type in the box
• Type in the box -> click on the button
• Type in the box -> click on the button -> submit
• Type in the box -> click on the button -> submit -> click a link in confirmation
• How often do users use the help pane more than once?
• “Use”: clicks a link or asks a question.
• Does the presence of the help pane lead to a lower probability of abandoning an edit session? (time to help) (secondary: it might increase time to abandonment)
• What percent of questions get answered?
• How quickly do questions get answered?
• What percent of users return to view the answer to their question? (extend EditorJourney beyond 24 hours for help desk? Or log only if someone has asked a question on the help desk)?
• What kind of questions to people ask?
• What kind of questions do and do not get answered?
• What percent of users had already read some type of help content before using the help pane?

• Do something before clicking
• Click on CTA
• Click on one of the help links
• Type some text in the question box
• Click “Ask a question”
• Submit
• Click on a link in the confirmation dialogue
• Close / collapse
• Do something afterwards.

What was the user doing (or attempting to do) right before interacting with the feature?
How often do users open the feature?
How often do users dismiss the feature?
If there are multiple options inside the feature, how often are those used?

In T206719#4762491, @JTannerWMF wrote:

Product and Analytics met to develop the following specifications:
Possible caveat: if the help pane makes it easier for a user to make their first edit, they are arguably being activated, thus we’re altering the probability of being activated. As a result, we might want to modify activation to only concern edits in the Main namespace.

Good catch. I think the pane would also be useful on Categories (to edit them) or on Help pages (juste because they may have a question about how to help others). For communities having those, on Portals.

• Does the presence of the help pane lead to questions being asked and answered?

It would certainly be possible if each question was a separate page, not a section potentially archived.

All questions about measuring the interaction on the questions are very interesting.

• What kind of questions do and do not get answered?

That one would be nice to have, so that we can see what blocks experienced users (and if we can help)

The questions that were listed in this ticket have grown to become the full "Help Panel -- measurement specifications" document: https://docs.google.com/document/d/1rnk-eT092c3CHESBgHxwN0udiXRP8lqgbENWQ-1RhBE/edit#

We have approval from the Manager of Product Analytics, as required by the Instrumentation DACI.

In T206711#4835724, @kzimmerman wrote:

@MMiller_WMF @JTannerWMF I've reviewed with @nettrom_WMF and approved Research Questions (DACI B), Instrumentation (DACI C), and Schema Creation (E). Data Dictionary (H) doesn't yet exist.

I think that's everything - let me know if you need anything else!