Page MenuHomePhabricator

Help panel: experiments
Closed, ResolvedPublic

Description

This is the only project in 2018Q2 that is expected to increase activation or retention rates. Therefore, we will want to structure a controlled experiment in which some new account holders are given the feature to use and some are not. We will want to see its impact on their activation or retention.

There may be other, smaller experiments we want to run, especially if we're considering different implementations of the feature's user experience. We might want to learn which one attracts more interaction, for instance.

Proposed experiments

A. Help panel on/off for new account holders
Goal: Determine the impact of the help panel on retention and activation

B. Compare a Help panel with the 5 help links and the "Ask a question" form vs one that only has the "Ask a question" form.

  • Goal/s:
    • Ascertain whether users are more likely to ask a question when no self-guided help is provided
    • Compare the quality of the questions in both versions
    • See if the differences in functionality availability impacts editor retention and activation

C. Compare a Help panel that allows posting a question to the help desk directly via the embedded "Ask a question" form, vs one that opens the Help Desk in a new tab instead.

  • Goal/s:
    • Determine whether users are more or less likely to post a question to the Help desk via the Help panel or directly
    • Compare the quality of the questions in both versions
    • See if the differences in functionality availability impacts editor retention and activation

D. Compare a help panel with strong affordance to one without

  • Goal/s:
    • Determine whether users are more or less likely to open and interact with a help panel that has some kind of pulsing or otherwise attractive UI affordance.
  • See if the differences in functionality availability impacts editor retention and activation

Event Timeline

mpopov moved this task from Triage to Backlog on the Product-Analytics board.Oct 26 2018, 6:34 PM
mpopov triaged this task as Medium priority.Oct 26 2018, 6:37 PM
MMiller_WMF renamed this task from Help desk: experiments to Help pane: experiments.Nov 13 2018, 11:31 PM
JTannerWMF reassigned this task from nettrom_WMF to RHo.Nov 20 2018, 3:46 PM

What if anything do you want to experiment with in this feature?
What are our primary goals what are our secondary goals?

RHo updated the task description. (Show Details)Nov 28 2018, 6:38 PM
RHo reassigned this task from RHo to MMiller_WMF.Nov 28 2018, 6:41 PM

hi @MMiller_WMF – I posted some potential experiments in the task description to run based on our chat earlier, please review and make additions/changes as you like.

JTannerWMF reassigned this task from MMiller_WMF to RHo.Nov 28 2018, 7:17 PM
JTannerWMF reassigned this task from RHo to MMiller_WMF.
MMiller_WMF renamed this task from Help pane: experiments to Help panel: experiments.Nov 29 2018, 7:15 PM
MMiller_WMF updated the task description. (Show Details)
MMiller_WMF updated the task description. (Show Details)

Thanks, @RHo. I added one additional experiment that I could think of.

The next step is for @nettrom_WMF to think this through and propose a plan for learning some things on a reasonable timeline.

I had a look at this and have a bunch of questions and thoughts that we should discuss and make decisions on.

  1. I think we are unlikely to be able to detect differences in retention rate between variants of the help panel, with potentially the exception of A/B-testing having the panel or not (but there we might only be able to detect changes in short-term retention, ref our many discussions of statistical power in this setting). This is because of the level of change in retention rate that will be required to be detectable on these wikis within a reasonable amount of time. We might instead be able to detect differences in leading indicators that might be connected to activation/retention (e.g. asking a question leads to an answer leads to a constructive edit leads to a retained user).
  2. If we give the feature to everyone we will run into self-selection issues. We won't be able to know if the help panel caused a change in activation/retention, or whether it was something about the users who chose to utilize the feature. Therefore, if we're interested in new editor retention. we either have to have a control group of those or figure out a clever way to get around this problem.
  3. Two of the key questions proposed in our draft measurement document is the panel's effect on activation and retention. We should think carefully about what features we think will have an effect on that, and which ones will not, and whether we think it's worth testing that.
  4. Why and how do we measure question quality? The reason I ask is that we do not appear to have any features that focus on question quality, e.g. seeking to have users ask "better" questions or guide the user through asking the question.

I'll think some more about this and come back with a more concrete proposal tomorrow.

To capture the further discussions we had about this today, the current proposal is:

Run an A/B test with new users where 50% have the feature turned off, and 50% have it turned on. Users in either group can turn the feature on/off through their preferences, something we will then account for in analysis (it is captured by the PrefUpdate schema). The goal of this A/B test is to determine if the help panel makes a significant impact on new user activation and retention.

I don't foresee that the proposed variants will have significant effects on activation/retention (within any reasonable timeline), because they all contain the same primary feature (shortcuts to the help desk). Therefore, we should be able to randomly assign users to variants in order to learn how the variants impact user behavior. We will need to define a specific set of measures for each variant in order to be able to do so.

One thing I'd be interested to experiment with is showing the smaller, mobile CTA in desktop context (the circle with question mark) vs the larger one that has text. I would be interested to see how that impacts usage of the tool and disabling of the tool via preferences.

nettrom_WMF moved this task from Backlog to Doing on the Product-Analytics board.Dec 5 2018, 10:50 PM
nettrom_WMF set Due Date to Jul 11 2019, 7:00 AM.Feb 13 2019, 7:04 PM
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptFeb 13 2019, 7:04 PM

Now that the help panel has been deployed for five weeks, we should have enough data to run the analysis for the activation experiment. Discussed with @nettrom_WMF today that this is a current priority.

@nettrom_WMF -- now that our team has discussed the results from the activation experiment, we need to post the results on wiki. That's the next action item on this task. Let's discuss priorities in the next few days.

Documenting that we've published the results in mediawiki.org, see this analytics update. As mentioned there, we're working on further analysis grouping data from all wikis together.

@nettrom_WMF: Should the status of this task still be "open", and the Priority field value be "normal"? Its deadline is set to "Jul 11 2019".

@Aklapper : thanks for the ping on this! This task should still be open as we've got a couple of related tasks open, once they are resolved I'll make sure to close this task as well. I've removed the deadline date since that's no longer relevant.

nettrom_WMF closed this task as Resolved.Apr 10 2020, 4:57 PM

The Growth Team has pivoted from Help Panel experiments to Newcomer Homepage experiments, so this task is no longer necessary to keep around, partly because we have a sub-task for the experiment analysis. Closing this for now, it can be reopened in the future if needed, or we'll create new tasks.