Page MenuHomePhabricator

Provide capability for A/B testing task types
Closed, ResolvedPublic

Description

Per the parent task:

In this A/B test setup, a randomly selected half of users who get the Growth features will get Add a Link tasks, and the other randomly selected half will get unstructured link tasks. We prefer to give users maximum exposure to these tasks and will therefore not give these users any copyedit tasks by default. In other words, for the purposes of this experiment, we’ll change the default difficulty filters from “links” and “copyedit” to just “links”. For wikis that don’t have unstructured link tasks, all those users get “add a link” and in that case we’ll exclude that wiki from the experiment.

Our code so far has assumed that all defined task types are available to all users. Here we will want to make the necessary changes so that we can A/B test providing the structured link task (link-recommendation) type to some users and the unstructured links task (links) to others.

Event Timeline

Also add a query flag for forcing a specific variant, like we had for the previous A/B task.

Change 678437 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] [WIP] Provide capability for A/B testing task types, step 1

https://gerrit.wikimedia.org/r/678437

Change 678438 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] [WIP] Provide capability for A/B testing task types, step 2

https://gerrit.wikimedia.org/r/678438

The user preference for being opted in to link recommendations is growthexperiments-link-recommendations-enabled. An example code snippet to set it is

new mw.Api().saveOption( 'growthexperiments-link-recommendations-enabled', 1 ).done( function() { window.location.reload() } );

Alternatively, you can use the geForceLinkRecommendationsEnabled query parameter during account creation to force the flag on or off.
When unforced, $wgGENewcomerTasksLinkRecommendationsEnablePercentage percent of new users who have their homepage enabled (ie. passed the $wgGEHomepageNewAccountEnablePercentage check) will be opted in.

kostajh triaged this task as Medium priority.Apr 14 2021, 12:53 PM

Change 681309 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] Update $wgGEHomepageNewAccountVariants

https://gerrit.wikimedia.org/r/681309

The user preference for being opted in to link recommendations is growthexperiments-link-recommendations-enabled. An example code snippet to set it is

new mw.Api().saveOption( 'growthexperiments-link-recommendations-enabled', 1 ).done( function() { window.location.reload() } );

Alternatively, you can use the geForceLinkRecommendationsEnabled query parameter during account creation to force the flag on or off.
When unforced, $wgGENewcomerTasksLinkRecommendationsEnablePercentage percent of new users who have their homepage enabled (ie. passed the $wgGEHomepageNewAccountEnablePercentage check) will be opted in.

@Tgr simplified this further to:

window.ge.utils.setUserVariant( 'linkrecommendation' )

and when you want to switch away from that variant:

window.ge.utils.setUserVariant( 'null' )

@Etonkovidova there are a lot of moving parts here, including code that will run for all newcomer users when this is in production, so prioritizing QA of this would be welcome. Some things to double-check:

  • legacy users with old variants (e.g. 'D', 'C' or 'A' or undefined/null) don't experience any breakages (not expected)
  • the geForceVariant flag in account creation works as intended
  • the window.ge.utils.setUserVariant( 'linkrecommendation' ) snippet works as intended and users see only the link-recommendation task type and not the links one

Change 678437 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Provide capability for A/B testing task types, step 1

https://gerrit.wikimedia.org/r/678437

Change 681309 merged by jenkins-bot:

[operations/mediawiki-config@master] Update $wgGEHomepageNewAccountVariants

https://gerrit.wikimedia.org/r/681309

Mentioned in SAL (#wikimedia-operations) [2021-04-21T18:38:49Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: f6d076a69607172475a86ba935a273e7519108d1: Update $wgGEHomepageNewAccountVariants (T278123) (duration: 00m 58s)

Change 678438 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Provide capability for A/B testing task types, step 2

https://gerrit.wikimedia.org/r/678438

Change 684378 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] GrowthExperiments: Set default variant

https://gerrit.wikimedia.org/r/684378

Change 684378 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Set default variant

https://gerrit.wikimedia.org/r/684378

Mentioned in SAL (#wikimedia-operations) [2021-05-03T12:33:21Z] <kharlan@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:684378|GrowthExperiments: Set default variant (T278123)]] [[gerrit:684331|GrowthExperiments: enable link recommendations frontend on cswiki (T278710)]] (duration: 00m 57s)

@Etonkovidova there are a lot of moving parts here, including code that will run for all newcomer users when this is in production, so prioritizing QA of this would be welcome. Some things to double-check:

  • legacy users with old variants (e.g. 'D', 'C' or 'A' or undefined/null) don't experience any breakages (not expected)
  • the geForceVariant flag in account creation works as intended
  • the window.ge.utils.setUserVariant( 'linkrecommendation' ) snippet works as intended and users see only the link-recommendation task type and not the links one

Checked all of the above - seems to be working as expected.