Page MenuHomePhabricator

Newcomer tasks: align treatment groups
Open, NormalPublic

Description

The homepage and help panel features are both running in A/B tests with half of newcomers having each one -- but the two tests are independent. So right now, 25% of newcomers have both features, 50% have either one or the other, and 25% have neither.

We intend for the newcomer tasks work to use both the homepage and help panel platforms. Therefore, we need to align the treatment groups for the homepage and help panel features so that all newcomers in treatment groups who get one also have the other, and the control group gets neither.

Here's how we want to do it:

Retroactively turn on the help panel for all users who had the homepage. Then 50% of old users (starting in the homepage era) will have both features. Also, 25% of old users will have help panel and no homepage. Going forward, align the treatment groups so that all new accounts in a treatment group get both features. Users that have both features then get newcomer tasks.

Event Timeline

MMiller_WMF created this task.
MMiller_WMF edited projects, added Growth-Team (Current Sprint); removed Growth-Team.

@nettrom_WMF @kostajh @Catrope @Tgr @marcella -- the first step with this one is to get some thoughts and ideas down around the right way to do it, and when the right time is to do it. I think we probably need to hear from @nettrom_WMF first because of the reverberations this change would have in our experiments that are currently running.

I think what we do here depends on whether we see Newcomer Tasks as a new experiment or a continuation of the existing Homepage experiment. If we see it as a continuation, then I'm primarily concerned about the users who are currently in the Homepage treatment group, and I'd then turn on the Help Panel for all of those so they can get the new features.

If Newcomer Tasks is a new experiment, and perhaps one where 100% of users get the Homepage so we can A/B test specific features of Newcomer Tasks, then I see alignment of preferences for previous users as answering two questions: 1) what's needed to ensure the Homepage works for the existing group?, and 2) should any of these features be activated for existing users? I'm not sure what the right answer to question 2 is, but as for 1 I think it's the same as earlier, we'll want to turn the Help Panel on so they can get Newcomer Tasks working correctly.

Do we have a separate phab tasks for discussing whether Newcomer Tasks is a new experiment or not?

@nettrom_WMF and I discussed this, and our current favorite idea is to split up the upcoming newcomer tasks experiment into three groups:

  • A: no homepage
  • B: homepage with one version of newcomer tasks
  • C: homepage with another version of newcomer tasks

This would allow us to see the overall impact of our work, as well as allow us to iterate on newcomer tasks.

@nettrom_WMF is now thinking through that approach and whether it is the best way forward.

Tgr added a comment.Mon, Sep 30, 4:27 PM

How are notices expected to work? E.g. if someone who got the help panel but not the homepage is now realigned to have both, do they have to see the homepage discovery tour?

@Tgr -- thanks for thinking about this. I don't think we would end up giving homepage to users that didn't start with it. It's more likely that we would do it the other way around: giving help panel to users that didn't start with it. We would not want users who are not brand new to start getting popups about their homepage. More to come on this soon as @nettrom_WMF and I finish drafting the measurement plan.

One aspect around this that has come up during the work on the Newcomer Tasks measurement plan is how we handle A/B testing variants of the intervention for existing users. We'll be discussing that in the team and see what we end up with.

nettrom_WMF triaged this task as Normal priority.Mon, Oct 7, 8:02 PM
nettrom_WMF moved this task from Triage to Doing on the Product-Analytics board.
MMiller_WMF updated the task description. (Show Details)Tue, Oct 15, 7:29 PM

@nettrom_WMF and I have decided what approach we want to take for aligning the treatment groups, and have updated the description in this task accordingly. This is ready for development. Because the help panel experiment is complete and the data is being pulled for the homepage experiment, it is okay to deploy this change when it is ready. It does not need to be coordinated with any other deployments of features.

nettrom_WMF removed nettrom_WMF as the assignee of this task.Tue, Oct 15, 7:32 PM