Page MenuHomePhabricator

Variant tests: align treatment groups
Closed, ResolvedPublic

Description

The homepage and help panel features are both running in A/B tests with half of newcomers having each one -- but the two tests are independent. So right now, 25% of newcomers have both features, 50% have either one or the other, and 25% have neither.

We intend for the newcomer tasks work to use both the homepage and help panel platforms. Therefore, we need to align the treatment groups for the homepage and help panel features so that all newcomers in treatment groups who get one also have the other, and the control group gets neither.

Here's how we want to do it:

Retroactively turn on the help panel for all users who had the homepage. Then 50% of old users (starting in the homepage era) will have both features. Also, 25% of old users will have help panel and no homepage. Going forward, align the treatment groups so that all new accounts in a treatment group get both features. Users that have both features then get newcomer tasks.

Event Timeline

MMiller_WMF created this task.
MMiller_WMF edited projects, added Growth-Team (Current Sprint); removed Growth-Team.

@nettrom_WMF @kostajh @Catrope @Tgr @marcella -- the first step with this one is to get some thoughts and ideas down around the right way to do it, and when the right time is to do it. I think we probably need to hear from @nettrom_WMF first because of the reverberations this change would have in our experiments that are currently running.

I think what we do here depends on whether we see Newcomer Tasks as a new experiment or a continuation of the existing Homepage experiment. If we see it as a continuation, then I'm primarily concerned about the users who are currently in the Homepage treatment group, and I'd then turn on the Help Panel for all of those so they can get the new features.

If Newcomer Tasks is a new experiment, and perhaps one where 100% of users get the Homepage so we can A/B test specific features of Newcomer Tasks, then I see alignment of preferences for previous users as answering two questions: 1) what's needed to ensure the Homepage works for the existing group?, and 2) should any of these features be activated for existing users? I'm not sure what the right answer to question 2 is, but as for 1 I think it's the same as earlier, we'll want to turn the Help Panel on so they can get Newcomer Tasks working correctly.

Do we have a separate phab tasks for discussing whether Newcomer Tasks is a new experiment or not?

@nettrom_WMF and I discussed this, and our current favorite idea is to split up the upcoming newcomer tasks experiment into three groups:

  • A: no homepage
  • B: homepage with one version of newcomer tasks
  • C: homepage with another version of newcomer tasks

This would allow us to see the overall impact of our work, as well as allow us to iterate on newcomer tasks.

@nettrom_WMF is now thinking through that approach and whether it is the best way forward.

Tgr added a comment.Sep 30 2019, 4:27 PM

How are notices expected to work? E.g. if someone who got the help panel but not the homepage is now realigned to have both, do they have to see the homepage discovery tour?

@Tgr -- thanks for thinking about this. I don't think we would end up giving homepage to users that didn't start with it. It's more likely that we would do it the other way around: giving help panel to users that didn't start with it. We would not want users who are not brand new to start getting popups about their homepage. More to come on this soon as @nettrom_WMF and I finish drafting the measurement plan.

One aspect around this that has come up during the work on the Newcomer Tasks measurement plan is how we handle A/B testing variants of the intervention for existing users. We'll be discussing that in the team and see what we end up with.

nettrom_WMF triaged this task as Medium priority.Oct 7 2019, 8:02 PM
nettrom_WMF moved this task from Triage to Doing on the Product-Analytics board.
MMiller_WMF updated the task description. (Show Details)Oct 15 2019, 7:29 PM

@nettrom_WMF and I have decided what approach we want to take for aligning the treatment groups, and have updated the description in this task accordingly. This is ready for development. Because the help panel experiment is complete and the data is being pulled for the homepage experiment, it is okay to deploy this change when it is ready. It does not need to be coordinated with any other deployments of features.

nettrom_WMF removed nettrom_WMF as the assignee of this task.Oct 15 2019, 7:32 PM

We've decided that aligning the treatment groups is not required for v1.0 because the help panel will not be part of the experience yet.

Moving off sprint board in favor of Newcomer Tasks V1.0 tasks.

MMiller_WMF moved this task from Inbox to Upcoming Work on the Growth-Team board.Nov 13 2019, 7:50 AM
MMiller_WMF renamed this task from Newcomer tasks: align treatment groups to Variant tests: align treatment groups.Nov 22 2019, 1:01 AM

Change 553218 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/extensions/GrowthExperiments@master] Allow auto-enable for help panel to be tied to homepage

https://gerrit.wikimedia.org/r/553218

Change 553222 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] GrowthExperiments: Align help panel new account enabling with homepage

https://gerrit.wikimedia.org/r/553222

Change 553218 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Allow auto-enable for help panel to be tied to homepage

https://gerrit.wikimedia.org/r/553218

Change 555639 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] [beta] Align GrowthExperiments treatment groups in beta labs

https://gerrit.wikimedia.org/r/555639

Change 555639 merged by jenkins-bot:
[operations/mediawiki-config@master] [beta] Align GrowthExperiments treatment groups in beta labs

https://gerrit.wikimedia.org/r/555639

Checked in betalabs - new users do get both of the features.
@MMiller_WMF - only two questions

  • I did not see old users got Help panel enabled. Would it be re-enabled for users who disabled it manually?
  • growthexperiments-tour-homepage-welcome has 0 for accounts that have growthexperiments-homepage-suggestededits-activated. I did not see any implications of it - welcome tour cannot be triggered by a user action.

@Catrope -- how will your implementation handle users who change their preferences manually? Does it, like, automatically turn off help panel if a user turns off homepage? Or is more like it sets them both on for a user, but then whatever happens after that is up to the user?

@Etonkovidova -- can you explain your second point? What are those phrases (growthexperiments-tour-homepage-welcome and growthexperiments-homepage-suggestededits-activated)?

@Etonkovidova -- can you explain your second point? What are those phrases (growthexperiments-tour-homepage-welcome and growthexperiments-homepage-suggestededits-activated)?

These growthexperiments-tour-homepage-welcome and growthexperiments-homepage-suggestededits-activated refers to db user_properties table. I actually re-checked in betalabs if users have homepage welcome tour marked as zero but suggested edits were activated - all seems to be fine.

@Catrope -- how will your implementation handle users who change their preferences manually? Does it, like, automatically turn off help panel if a user turns off homepage? Or is more like it sets them both on for a user, but then whatever happens after that is up to the user?

It's the latter: at signup time, we will either enable both or neither, but after that the user can do whatever they want in their preferences. There are still separate preferences for the homepage and the help panel, and the user can choose to turn on/off one but not the other.

Change 553222 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: Align help panel new account enabling with homepage

https://gerrit.wikimedia.org/r/553222

Mentioned in SAL (#wikimedia-operations) [2019-12-13T00:10:58Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Align help panel new account enabling with homepage (T232396) (duration: 00m 56s)

MMiller_WMF closed this task as Resolved.Dec 13 2019, 11:55 PM

In production as of 2019-12-12 (pacific time). Thank you!

Urbanecm edited subscribers, added: Urbanecm_WMF; removed: Urbanecm.Aug 26 2020, 3:24 PM