This came up recently in a separate context, but I can't find a link at the moment. AIUI, it is intentional that new users are intentionally placed into an experiment or control group with three separate experiments:

• WelcomeSurvey (only on arwiki is this an A/B test with 50% of new users in control and 50% in experiment, on cswiki and kowiki it's 100%)
• Homepage (50% of new users in control, 50% in experiment)
• Help panel (50% of new users in control, 50% in experiment)

@nettrom_WMF @MMiller_WMF please close this task unless you want to add anything here or change the business rules for the experiments.

As Kosta mentions, this is currently by design since we're experimenting with these features in order to understand their effect (with a focus on editor activation and retention). Prior to launching the Help Panel, we did discuss and carefully consider whether to launch features in combination or separately. One of the concerns was that maybe some features only work in combination, and if we launch them together we maximize the probability of that. However, we would then be unable to learn whether a feature works by itself or whether it's the combination that works. So, we've chosen to keep assigning users randomly to experiment/control groups until we know more about their effects.

Once we have results of our analysis, we can consider whether to push the features to everyone or not. We've already done that with the Welcome Survey, when we saw there were no significant negative effects on neither Czech nor Korean Wikipedia we switched so everyone got the survey.

I'm unsure whether to close this task or not, so I'll leave it to @MMiller_WMF to decide.

@Urbanecm -- thanks for bringing this up. It didn't occur to me that mentors might be pointing users to features that they don't have. Taken alone, I don't think that is enough of a reason to change our experiment.

But we may end up wanting to change based on work we'll be doing with newcomer tasks. We're considering using the help panel to guide newcomers while they are doing a recommended task, and in that case, newcomers with the homepage would also need to have the help panel. That might be a turning point in which we put users into the same treatment group for both those features.

@kostajh @Catrope @Tgr -- just wondering off the tops of your heads whether it will be trivial to put users in both the help panel and homepage treatment/control groups, or whether we would need to spend time changing how we assign users to experiment conditions.

just wondering off the tops of your heads whether it will be trivial to put users in both the help panel and homepage treatment/control groups, or whether we would need to spend time changing how we assign users to experiment conditions.

Yes, this doesn't seem like it would be very difficult to do.