I had a look at this and have a bunch of questions and thoughts that we should discuss and make decisions on.

1. I think we are unlikely to be able to detect differences in retention rate between variants of the help panel, with potentially the exception of A/B-testing having the panel or not (but there we might only be able to detect changes in short-term retention, ref our many discussions of statistical power in this setting). This is because of the level of change in retention rate that will be required to be detectable on these wikis within a reasonable amount of time. We might instead be able to detect differences in leading indicators that might be connected to activation/retention (e.g. asking a question leads to an answer leads to a constructive edit leads to a retained user).
2. If we give the feature to everyone we will run into self-selection issues. We won't be able to know if the help panel caused a change in activation/retention, or whether it was something about the users who chose to utilize the feature. Therefore, if we're interested in new editor retention. we either have to have a control group of those or figure out a clever way to get around this problem.
3. Two of the key questions proposed in our draft measurement document is the panel's effect on activation and retention. We should think carefully about what features we think will have an effect on that, and which ones will not, and whether we think it's worth testing that.
4. Why and how do we measure question quality? The reason I ask is that we do not appear to have any features that focus on question quality, e.g. seeking to have users ask "better" questions or guide the user through asking the question.

I'll think some more about this and come back with a more concrete proposal tomorrow.

To capture the further discussions we had about this today, the current proposal is:

Run an A/B test with new users where 50% have the feature turned off, and 50% have it turned on. Users in either group can turn the feature on/off through their preferences, something we will then account for in analysis (it is captured by the PrefUpdate schema). The goal of this A/B test is to determine if the help panel makes a significant impact on new user activation and retention.

I don't foresee that the proposed variants will have significant effects on activation/retention (within any reasonable timeline), because they all contain the same primary feature (shortcuts to the help desk). Therefore, we should be able to randomly assign users to variants in order to learn how the variants impact user behavior. We will need to define a specific set of measures for each variant in order to be able to do so.

One thing I'd be interested to experiment with is showing the smaller, mobile CTA in desktop context (the circle with question mark) vs the larger one that has text. I would be interested to see how that impacts usage of the tool and disabling of the tool via preferences.

Now that the help panel has been deployed for five weeks, we should have enough data to run the analysis for the activation experiment. Discussed with @nettrom_WMF today that this is a current priority.

@nettrom_WMF -- now that our team has discussed the results from the activation experiment, we need to post the results on wiki. That's the next action item on this task. Let's discuss priorities in the next few days.

Documenting that we've published the results in mediawiki.org, see this analytics update. As mentioned there, we're working on further analysis grouping data from all wikis together.

@nettrom_WMF: Should the status of this task still be "open", and the Priority field value be "normal"? Its deadline is set to "Jul 11 2019".

@Aklapper : thanks for the ping on this! This task should still be open as we've got a couple of related tasks open, once they are resolved I'll make sure to close this task as well. I've removed the deadline date since that's no longer relevant.