Page MenuHomePhabricator
Create Task
Maniphest T377713

Do not call ExperimentUserManager::setVariant on all newly registered accounts
Closed, ResolvedPublic
Actions

Description

Since the introduction of conditional user options as the underlying mechanism for assigning an experiment variant (T376266), the existing code in GrowthExperiments that assigns a variant to newly created accounts (HomepageHooks.php#882) becomes redundant or "competing" with the conditionally default assignment. This task is to stop calling unconditionally setVariant/setOption when a new account is created in a wiki with GrowthExperiments.

Acceptance criteria

  • No variant is stored in the user_properties table for the preference growthexperiments-homepage-variant, except for forced via URL (acceptance criteria 2) or via ge.utils.setUserVariant()
  • geForceVariant parameter overrides the variant assigned via defaults

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
HomepageHooks: do not store assigned variant on account creationmediawiki/extensions/GrowthExperimentswmf/1.43.0-wmf.28+13 -123
HomepageHooks: do not store assigned variant on account creationmediawiki/extensions/GrowthExperimentswmf/1.44.0-wmf.1+13 -123
HomepageHooks: do not store assigned variant on account creationmediawiki/extensions/GrowthExperimentsmaster+13 -123
Customize query in gerrit

Related Objects

Event Timeline

Sgs created this task.Oct 21 2024, 11:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 21 2024, 11:59 AM
Sgs added a project: GrowthExperiments.Oct 21 2024, 11:59 AM
Sgs updated the task description. (Show Details)
Urbanecm_WMF mentioned this in T377631: Add a link (Structured task): Release to a subset of newcomers on English Wikipedia.Oct 21 2024, 1:53 PM
Sgs claimed this task.Oct 24 2024, 3:10 PM
Sgs edited projects, added Growth-Team (Current Sprint); removed Growth-Team.
Sgs moved this task from Incoming to Doing on the Growth-Team (Current Sprint) board.
gerritbot added a comment.Oct 24 2024, 4:48 PM

Change #1082816 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] HomepageHooks: stop assigning variants on account creation

https://gerrit.wikimedia.org/r/1082816

gerritbot added a project: Patch-For-Review.Oct 24 2024, 4:48 PM
Sgs moved this task from Doing to Code Review on the Growth-Team (Current Sprint) board.Oct 25 2024, 11:40 AM
Sgs updated the task description. (Show Details)Oct 25 2024, 3:04 PM
Sgs added subscribers: KStoller-WMF, Urbanecm_WMF, Iflorez, nettrom_WMF.Oct 28 2024, 12:19 PM

It's been brought up in code review that if we stop assigning a (different) variant than what conditional defaults already provides (T376266) to newly created accounts, it does not make sense to keep infrastructure (code and configuration) that backs it. Otoh if we do remove this mechanism, conditional defaults becomes our only way for setting up experiments. Per prior conversations with @nettrom_WMF and @KStoller-WMF I believe this is ok, but I'm looking for a confirmation. Looping @Iflorez and @Urbanecm_WMF as well.

The main difference is that we won't have the possibility to conduct an experiment "as we did in the past". Where this mainly means three aspects:

  • Users were only enrolled into experiments on account creation
  • Configuration allowed per-platform distribution (although it's been evaluated as a trick to do feature flagging on a non-ready platform)
  • Side effect: a percentage of newly created accounts occupies space in DB that later needs to be cleaned

I believe it is ok to move on with a system that does not differentiate between existing and new accounts, I believe filtering results by each of these groups of users can be done when doing the experiment analysis filtering by registration date, but confirmation from @nettrom_WMF or @Iflorez is appreciated.
The per-platform distribution configuration was used in the past to avoid enrolling users from an specific platform into the experiment because that feature had not yet been developed for the platform. Although this seems handy, one is not to expect any instrumentation data/traffic from a non-delivered feature so not sure how useful this is. It depends on how we count "things", if variant assignments are counted (they are in grafana dashboards) for something in the experiment analysis it could matter but otherwise, if we only count homepage visits from the platform we know the feature is in, it should not be a problem.

Sgs moved this task from Code Review to QA on the Growth-Team (Current Sprint) board.Oct 29 2024, 4:49 PM
gerritbot added a comment.Oct 29 2024, 5:22 PM

Change #1084181 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.1] HomepageHooks: do not store assigned variant on account creation

https://gerrit.wikimedia.org/r/1084181

gerritbot added a comment.Oct 30 2024, 12:20 PM

Change #1082816 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] HomepageHooks: do not store assigned variant on account creation

https://gerrit.wikimedia.org/r/1082816

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.2; 2024-11-05).Oct 30 2024, 1:00 PM
gerritbot added a comment.Oct 30 2024, 8:30 PM

Change #1084181 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.1] HomepageHooks: do not store assigned variant on account creation

https://gerrit.wikimedia.org/r/1084181

ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.1; 2024-10-29); removed MW-1.44-notes (1.44.0-wmf.2; 2024-11-05).Oct 30 2024, 9:00 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 30 2024, 9:30 PM
gerritbot added a comment.Oct 31 2024, 10:05 AM

Change #1085346 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@wmf/1.43.0-wmf.28] HomepageHooks: do not store assigned variant on account creation

https://gerrit.wikimedia.org/r/1085346

gerritbot added a project: Patch-For-Review.Oct 31 2024, 10:05 AM
gerritbot added a comment.Oct 31 2024, 1:26 PM

Change #1085346 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.43.0-wmf.28] HomepageHooks: do not store assigned variant on account creation

https://gerrit.wikimedia.org/r/1085346

Stashbot mentioned this in T378573: Wikibase CI blocked by errors in SkinAfterPortletHandlerTest and ChangesListSpecialPageHookHandlerTest.Oct 31 2024, 1:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-10-31T13:28:03Z] <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1085342|Set username in user mock and reset state after test (T378573)]], [[gerrit:1085343|Fix and re-enable selenium test (T378581)]], [[gerrit:1085344|Fix selenium test loading the wrong talk page]], [[gerrit:1085346|HomepageHooks: do not store assigned variant on account creation (T377713)]], [[gerrit:1085347|SpecialHomepage: show community update

Stashbot mentioned this in T378581: Re-enable browser test in repo/tests/selenium/specs/item.js.Oct 31 2024, 1:28 PM
Stashbot added a comment.Oct 31 2024, 1:30 PM

Mentioned in SAL (#wikimedia-operations) [2024-10-31T13:30:36Z] <urbanecm@deploy2002> hnowlan, sgimeno, urbanecm: Backport for [[gerrit:1085342|Set username in user mock and reset state after test (T378573)]], [[gerrit:1085343|Fix and re-enable selenium test (T378581)]], [[gerrit:1085344|Fix selenium test loading the wrong talk page]], [[gerrit:1085346|HomepageHooks: do not store assigned variant on account creation (T377713)]], [[gerrit:1085347|SpecialHomepage: show community upda

Maintenance_bot removed a project: Patch-For-Review.Oct 31 2024, 1:31 PM
Stashbot added a comment.Oct 31 2024, 1:38 PM

Mentioned in SAL (#wikimedia-operations) [2024-10-31T13:38:47Z] <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1085342|Set username in user mock and reset state after test (T378573)]], [[gerrit:1085343|Fix and re-enable selenium test (T378581)]], [[gerrit:1085344|Fix selenium test loading the wrong talk page]], [[gerrit:1085346|HomepageHooks: do not store assigned variant on account creation (T377713)]], [[gerrit:1085347|SpecialHomepage: show community update

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.28; 2024-10-22).Oct 31 2024, 2:00 PM
Etonkovidova closed this task as Resolved.Nov 5 2024, 11:44 PM
Etonkovidova updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits