Page MenuHomePhabricator

Users should have either both homepage and help panel, or none of those projects
Closed, DeclinedPublic

Description

On projects where both homepage and help panel are enabled, one should either have both homepage and help panel, or none of those features.

Mentors can advice users to use the other feature (ie. advice using help panel while answering to mentorship question), or the other way around.

Not sure if we currently do some analytics based on no homepage and help panel, and vica versa. @MMiller_WMF, could you comment on this? Otherwise, we can advice mentors against doing so in T224633: Growth: training for how to help newcomers.

Event Timeline

Urbanecm created this task.Aug 25 2019, 8:48 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptAug 25 2019, 8:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Urbanecm updated the task description. (Show Details)Aug 25 2019, 8:48 PM
kostajh added subscribers: nettrom_WMF, kostajh.

This came up recently in a separate context, but I can't find a link at the moment. AIUI, it is intentional that new users are intentionally placed into an experiment or control group with three separate experiments:

  • WelcomeSurvey (only on arwiki is this an A/B test with 50% of new users in control and 50% in experiment, on cswiki and kowiki it's 100%)
  • Homepage (50% of new users in control, 50% in experiment)
  • Help panel (50% of new users in control, 50% in experiment)

@nettrom_WMF @MMiller_WMF please close this task unless you want to add anything here or change the business rules for the experiments.

As Kosta mentions, this is currently by design since we're experimenting with these features in order to understand their effect (with a focus on editor activation and retention). Prior to launching the Help Panel, we did discuss and carefully consider whether to launch features in combination or separately. One of the concerns was that maybe some features only work in combination, and if we launch them together we maximize the probability of that. However, we would then be unable to learn whether a feature works by itself or whether it's the combination that works. So, we've chosen to keep assigning users randomly to experiment/control groups until we know more about their effects.

Once we have results of our analysis, we can consider whether to push the features to everyone or not. We've already done that with the Welcome Survey, when we saw there were no significant negative effects on neither Czech nor Korean Wikipedia we switched so everyone got the survey.

I'm unsure whether to close this task or not, so I'll leave it to @MMiller_WMF to decide.

@Urbanecm -- thanks for bringing this up. It didn't occur to me that mentors might be pointing users to features that they don't have. Taken alone, I don't think that is enough of a reason to change our experiment.

But we may end up wanting to change based on work we'll be doing with newcomer tasks. We're considering using the help panel to guide newcomers while they are doing a recommended task, and in that case, newcomers with the homepage would also need to have the help panel. That might be a turning point in which we put users into the same treatment group for both those features.

@kostajh @Catrope @Tgr -- just wondering off the tops of your heads whether it will be trivial to put users in both the help panel and homepage treatment/control groups, or whether we would need to spend time changing how we assign users to experiment conditions.

just wondering off the tops of your heads whether it will be trivial to put users in both the help panel and homepage treatment/control groups, or whether we would need to spend time changing how we assign users to experiment conditions.

Yes, this doesn't seem like it would be very difficult to do.

MMiller_WMF closed this task as Declined.Sep 17 2019, 11:10 PM

Thanks, @kostajh.

@Urbanecm -- we have decided to align treatment groups between the two features to facilitate the newcomer tasks project, which relies on users having both features. We're tracking and discussing that in T232396.