Page MenuHomePhabricator

Add a link: experiment
Closed, ResolvedPublic

Description

We have defined our research questions, measurements, experiment plan, and leading indicators in the "Add a Link - Measurement Specification". This task contains the experiment plan.


In this A/B test setup, a randomly selected half of users who get the Growth features will get Add a Link tasks, and the other randomly selected half will get unstructured link tasks. We prefer to give users maximum exposure to these tasks and will therefore not give these users any copyedit tasks by default. In other words, for the purposes of this experiment, we’ll change the default difficulty filters from “links” and “copyedit” to just “links”. For wikis that don’t have unstructured link tasks, all those users get “add a link” and in that case we’ll exclude that wiki from the experiment.

We plan to continue to have a Control group that does not get access to the Growth features, which is a randomly selected 20% of new registrations.

Group A: users get “add a link” as their only default task type.
Group B: users get unstructured link task as their only default task type.
Group C: control (no Growth features)

The experiment will run for a limited time, most likely between four to eight weeks. We base this duration on our most recent Variant C/D test, which had a similar setup of randomly assigned groups. That experiment ran for four weeks and the dataset contained 67,598 registrations. Since then, the Growth features have been deployed on several additional wikis, further reducing the need for experiments with a longer duration.

In practice the experiment will start with our four pilot wikis. After two weeks, we will analyze the leading indicators in T277355: Add a link: leading indicators to decide whether to extend the experiment to the rest of the Growth wikis.

Users who registered prior to the experiment and have access to the Growth features will have the unstructured link task replaced by Add a Link. These users will not be part of the experiment itself (because they were not randomly selected into treatment groups), and get the feature so that we can expose it to a larger group of contributors.

Event Timeline

kostajh triaged this task as Medium priority.Apr 15 2021, 1:00 PM
kostajh added a project: Add-Link.
kostajh moved this task from Backlog to May 3 - May 7 on the Add-Link board.

There's not anything "ready for development" here per se, so I'm inclined to either move this to "Epics" (but it doesn't really belong there), or remove from current sprint. I'll go with removing from current sprint. When we're ready to start the experiment we can move back to the current sprint board.

kostajh moved this task from Triaged to Upcoming Work on the Growth-Team board.
kostajh set Due Date to May 23 2021, 10:00 PM.
kostajh changed Due Date from May 23 2021, 10:00 PM to May 24 2021, 10:00 PM.May 10 2021, 5:16 PM

@kostajh -- this is the task that we'll resolve once we've deployed to users. The task description includes the deployment for new accounts, and for our legacy newcomers.

Change 695364 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] GrowthExperiments: Enable Add Links for 50% of new users and all old ones

https://gerrit.wikimedia.org/r/695364

Change 695364 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Enable Add Links for 50% of new users and all old ones

https://gerrit.wikimedia.org/r/695364

Mentioned in SAL (#wikimedia-operations) [2021-05-27T19:12:03Z] <tgr@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:695364|GrowthExperiments: Enable Add Links for 50% of new users and all old ones (T277356)]] (duration: 01m 04s)

Mentioned in SAL (#wikimedia-operations) [2021-05-27T19:12:03Z] <tgr@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:695364|GrowthExperiments: Enable Add Links for 50% of new users and all old ones (T277356)]] (duration: 01m 04s)

The experiment started at 2021-05-27T19:12:03Z on the pilot wikis (arwiki, bnwiki, cswiki, viwiki):

  • Group A: users get “add a link” as their only default task type (40%)
  • Group B: users get unstructured link task as their only default task type (40%)
  • Group C: control (no Growth features) (20%)

Or in other words – 20% get no Growth features, 80% get the homepage, of the homepage group, 50% get the new link suggestion task and 50% get the old unstructured link task.

Users who registered prior to the experiment and have access to the Growth features will have the unstructured link task replaced by Add a Link. These users will not be part of the experiment itself (because they were not randomly selected into treatment groups), and get the feature so that we can expose it to a larger group of contributors.

This went into effect with the exception of users registered in the month of May; these users have the variant 'control' associated with their account and as such were not automatically moved over to the 'linkrecommendation' variant. We will move those users into the linkrecommendation variant next week after we write the script needed to do that T283867: Maintenance script for changing user settings.