Page MenuHomePhabricator

Homepage: experiment
Open, NormalPublicTue, Oct 8

Description

This task is about setting up the A/B test for the newcomer homepage. In order to understand the homepage’s impact on editor activation and retention, we propose an experiment that will be running for at least six months. During that time, 50% of new registrations on target wikis will have the homepage enabled by default, along with the rest of the homepage treatment, and 50% will have it disabled. These 50% pieces should be independent of splits made for any other experiment (such as for help panel or welcome survey).

By “homepage treatment” we mean that a user both has the homepage on by default and is receiving the various ancillary features that drive them to the homepage, which may include banners, talk page messages, or links on Main Page. In the initial version, the only ancillary feature will be the fact that the username link in personal tools will go to the homepage. So in other words, when someone is in the treatment group, it may mean that they have several things turned on, and the control would have none of the things turned on.

We may also want to run deeper splits later on, such as splitting the treatment group (who get the newcomer homepage treatment) into two groups -- one that has a certain homepage configuration or set of treatments and another that doesn't.

When users are assigned to treatment and control groups, we will need to know which groups they are in, so they can be separated in the analysis.

See the "Experiment plan" section in the measurement specifications for full details.

When we analyze the experiment, we will want to look at these KPIs:

  • Activation
  • Retention
  • Edit count/volume
  • Productive edits (non-reverted)
  • Having a confirmed email address
  • Having created a user page (even if subsequently deleted)

Details

Due Date
Tue, Oct 8, 4:00 AM

Event Timeline

Hey @MMiller_WMF can you add more details to this?

MMiller_WMF updated the task description. (Show Details)Mar 27 2019, 12:23 AM
MMiller_WMF updated the task description. (Show Details)

@nettrom_WMF -- could you please look this over and make any changes? Then it will be ready for development.

Note the the code to enable the homepage preferences to X% of new local accounts is already in place. It was built as part of T215982: Homepage: location, navigation, and framework

When we're ready to start the experiment, a config patch will be needed to set $wgGEHomepageNewAccountEnablePercentage = 50;

@MMiller_WMF : this looks good to go to me!

When the homepage is deployed to Czech and Korean on 2019-05-02, we'll deploy at 0% of newcomers in the treatment group. That gives us and @Urbanecm and @revi time to test the homepage out in their wikis before newcomers get it. If everything looks good, we'll start the experiment at 50% treatment group on Monday, 2019-05-06.

JTannerWMF set Due Date to Nov 6 2019, 5:00 AM.
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptMay 8 2019, 4:20 PM

The experiment began on Korean and Czech Wikipedias at 19:01 UTC on May 6.

nettrom_WMF moved this task from Triage to Doing on the Product-Analytics board.
nettrom_WMF triaged this task as Normal priority.Jul 2 2019, 9:02 PM
MMiller_WMF changed Due Date from Nov 6 2019, 5:00 AM to Tue, Oct 8, 4:00 AM.Aug 10 2019, 12:49 AM
MMiller_WMF updated the task description. (Show Details)

@nettrom_WMF -- per our conversation, I added a date of October 7 to this as a target. We said we could pull data at the beginning of October, and that analyzing it would be faster and easier than doing it for help panel. We can adjust this date as needed.

I also listed specific KPIs, and included edit volume, productive edits, having a confirmed email, and creating user page as things we want to look at. We can discuss more as this date approaches.