Page MenuHomePhabricator
Create Task
Maniphest T416894

Integrate existing AB testable features with TestKitchen
Closed, ResolvedPublic
Actions

Description

Currently GrowthExperiments makes use of ExperimentUserManager or ExperimentUserDefaultsManager in a number of places (bespoke system) and also use of ExperimentTestKitchenManager in some others. We should move all experiment-able features to make use of the new system ExperimentTestKitchenManager. Or even more ambitiously, consider completely removing TestKitchen as a soft-dependency of GrowthExperiments and write all experiment code in WikimediaEvents.

To do so it would come very handy to have a product decision on each feature, should the feature remain AB-testable or should we get rid of the AB-testing setup for it?

Features affected

  • SiteNoticeGenerator
  • Task types (currently used for Revise Tone)
  • Leveling up notifications
  • Account creation section KPI dashboard

"Files" affected

  • SiteNoticeGenerator, SiteNoticeGeneratorTest
  • NewcomerTaskUserOptionsLookup, NewcomerTaskUserOptionsLookupTest (task type AB testing)
  • SuggestedEdits.php (support for Revise tone experiment) — stays as it is until experiment concludes
  • ExperimentsHooks (support for experiments using wgConditionalUserOptions)
  • HomepageHooks (support for Account creation section in Growth KPI dashboard, support for $geForceVariant )

Acceptance criteria

  • ExperimentUserManager is removed
  • ExperimentUserDefaultsManager is removed
  • Features that have been deemed not relevant to AB test anymore do not make use of any experiments manager

Details

Other Assignee
Cyndymediawiksim
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Experiments: prefer FeatureManager over ExprimentManagermediawiki/extensions/GrowthExperimentsmaster+219 -246
Experiments: dismantle ExperimentUserManagermediawiki/extensions/GrowthExperimentsmaster+88 -373
LevelingUpManager: stop supporting multiple delay specificationsmediawiki/extensions/GrowthExperimentswmf/1.46.0-wmf.16+4 -117
SiteNoticeGenerator: stop adding per-variant classesmediawiki/extensions/GrowthExperimentswmf/1.46.0-wmf.16+3 -31
ExperimentManager: remove geForceVariant flag handlingmediawiki/extensions/GrowthExperimentswmf/1.46.0-wmf.16+21 -89
LevelingUpManager: stop supporting multiple delay specificationsmediawiki/extensions/GrowthExperimentsmaster+4 -117
SiteNoticeGenerator: stop adding per-variant classesmediawiki/extensions/GrowthExperimentsmaster+3 -31
ExperimentManager: remove geForceVariant flag handlingmediawiki/extensions/GrowthExperimentsmaster+21 -89
ExperimentTestKitchenManager: stop enrolling users on account creationmediawiki/extensions/GrowthExperimentsmaster+15 -58
experiments: dismantle ExperimentUserDefaultsManagermediawiki/extensions/GrowthExperimentsmaster+1 -474
Customize query in gerrit

Related Objects
Search...

Event Timeline

Sgs created this task.Feb 9 2026, 4:45 PM
Sgs updated the task description. (Show Details)Feb 10 2026, 3:42 PM
Sgs updated the task description. (Show Details)Feb 10 2026, 3:55 PM
Sgs updated the task description. (Show Details)
Sgs updated the task description. (Show Details)Feb 10 2026, 3:58 PM
Sgs added subscribers: KStoller-WMF, Urbanecm_WMF.EditedFeb 10 2026, 4:52 PM

After a conversation with @Urbanecm_WMF about this we have agreed on some conclusions:

  • Account creation section KPI dashboard code can be removed, already confirmed by @KStoller-WMF and if we want to have some experiment exposure KPIs in Grafana we could it in T416583
  • Conditional defaults driven experiments are not gonna be used in the future and ExperimentUserDefaultsManager can be dismantled
  • We discussed about David's pattern in this thread and agreed it would be tricky to apply for server side rendered HTML as there's no easy way of transporting the enrollment result from WikimediaEvents to the relevant SpecialPage rendering code without a dedicated intervention, eg: registering some provider from extension.json. The best shot GrowthExperiments has to facade the feature toggling logic based on experiment variants is using the recently introduced FeatureManager
gerritbot added a comment.Feb 10 2026, 5:28 PM

Change #1238399 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] experiments: dismantle ExperimentUserDefaultsManager

https://gerrit.wikimedia.org/r/1238399

gerritbot added a project: Patch-For-Review.Feb 10 2026, 5:28 PM
Sgs triaged this task as Medium priority.Feb 10 2026, 5:45 PM
Sgs moved this task from Inbox to needs estimation on the Growth-Team board.
Sgs added a comment.Feb 10 2026, 6:19 PM

Removing ExperimentUserDefaultsManager leaves ConditionalDefaultOptionsAddConditionHook without any consumer, any thoughts @Urbanecm_WMF ?

Sgs updated the task description. (Show Details)Feb 10 2026, 6:41 PM
Sgs renamed this task from Decide how to integrate existing AB testable feature with TestKitchen to Integrate existing AB testable features with TestKitchen.Feb 10 2026, 6:44 PM
gerritbot added a comment.Feb 10 2026, 6:44 PM

Change #1238411 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] ExperimentTestKitchenManager: stop enrolling users on account creation

https://gerrit.wikimedia.org/r/1238411

gerritbot added a comment.Feb 11 2026, 11:25 AM

Change #1238399 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] experiments: dismantle ExperimentUserDefaultsManager

https://gerrit.wikimedia.org/r/1238399

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.16; 2026-02-17).Feb 11 2026, 12:00 PM
Urbanecm_WMF added a comment.Feb 11 2026, 4:44 PM
In T416894#11603638, @Sgs wrote:

Removing ExperimentUserDefaultsManager leaves ConditionalDefaultOptionsAddConditionHook without any consumer, any thoughts @Urbanecm_WMF ?

It was added in 1.43, which means it is covered by the Stable Interface Policy. Removing it would require first deprecating it, and then removing it no less than a release later. Since the hook is not causing any problems and given we aren't aware of any structural problems with it, I think it is best to leave it there. Happy to consider other arguments if I'm missing something though!

gerritbot added a comment.Feb 11 2026, 6:42 PM

Change #1238411 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] ExperimentTestKitchenManager: stop enrolling users on account creation

https://gerrit.wikimedia.org/r/1238411

Maintenance_bot removed a project: Patch-For-Review.Feb 11 2026, 7:31 PM
Sgs updated the task description. (Show Details)Mon, Feb 16, 3:19 PM
gerritbot added a comment.Mon, Feb 16, 5:26 PM

Change #1239752 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] ExperimentManager: remove geForceVariant flag handling

https://gerrit.wikimedia.org/r/1239752

gerritbot added a project: Patch-For-Review.Mon, Feb 16, 5:26 PM
Sgs mentioned this in T417621: Remove 'growthexperiments-homepage-variant' user property from all wikis where it's present.Tue, Feb 17, 9:33 AM
Cyndymediawiksim mentioned this in T415536: Allow running multiple experiments in GrowthExperiments at the same time.Tue, Feb 17, 9:34 AM
Aklapper added a parent task: T415536: Allow running multiple experiments in GrowthExperiments at the same time.Tue, Feb 17, 10:45 AM
Sgs claimed this task.Tue, Feb 17, 11:16 AM
Sgs updated Other Assignee, added: Cyndymediawiksim.
Sgs moved this task from needs estimation to FY2025-26 Q3 Sprint 3 on the Growth-Team board.
Sgs edited projects, added Growth-Team (FY2025-26 Q3 Sprint 3); removed Growth-Team.
gerritbot added a comment.Tue, Feb 17, 11:28 AM

Change #1239911 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] SiteNoticeGenerator: stop adding per-variant classes

https://gerrit.wikimedia.org/r/1239911

Sgs moved this task from Incoming to Doing on the Growth-Team (FY2025-26 Q3 Sprint 3) board.Tue, Feb 17, 11:39 AM
gerritbot added a comment.Tue, Feb 17, 11:54 AM

Change #1239752 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] ExperimentManager: remove geForceVariant flag handling

https://gerrit.wikimedia.org/r/1239752

ReleaseTaggerBot edited projects, added MW-1.46-notes (1.46.0-wmf.17; 2026-02-24); removed MW-1.46-notes (1.46.0-wmf.16; 2026-02-17).Tue, Feb 17, 12:00 PM
gerritbot added a comment.Tue, Feb 17, 8:20 PM

Change #1240039 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] LevelingUpManager: stop supporting multiple delay specifications

https://gerrit.wikimedia.org/r/1240039

gerritbot added a comment.Tue, Feb 17, 9:19 PM

Change #1239911 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] SiteNoticeGenerator: stop adding per-variant classes

https://gerrit.wikimedia.org/r/1239911

Sgs mentioned this in T375198: Fully adopt TestKitchen for experiment enrollment.Wed, Feb 18, 3:59 PM
gerritbot added a comment.Wed, Feb 18, 9:36 PM

Change #1240039 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] LevelingUpManager: stop supporting multiple delay specifications

https://gerrit.wikimedia.org/r/1240039

Maintenance_bot removed a project: Patch-For-Review.Wed, Feb 18, 10:31 PM
Sgs mentioned this in T417876: Migrate existing instruments that add experiment information to use TestKitchen variant.Thu, Feb 19, 12:02 PM
gerritbot added a comment.Thu, Feb 19, 1:26 PM

Change #1240693 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] Experiments: prefer FeatureManager over ExprimentManager

https://gerrit.wikimedia.org/r/1240693

gerritbot added a project: Patch-For-Review.Thu, Feb 19, 1:26 PM
gerritbot added a comment.Thu, Feb 19, 4:29 PM

Change #1239893 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] Experiments: dismantle ExperimentUserManager

https://gerrit.wikimedia.org/r/1239893

Urbanecm_WMF edited projects, added Growth-Team (FY2025-26 Q3 Sprint 4); removed Growth-Team (FY2025-26 Q3 Sprint 3).Tue, Feb 24, 5:40 PM
Urbanecm_WMF moved this task from Incoming to Doing on the Growth-Team (FY2025-26 Q3 Sprint 4) board.
DMburugu added a project: Essential-Work.Tue, Feb 24, 5:56 PM
gerritbot added a comment.Wed, Feb 25, 3:18 PM

Change #1243855 had a related patch set uploaded (by Urbanecm; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@wmf/1.46.0-wmf.16] ExperimentManager: remove geForceVariant flag handling

https://gerrit.wikimedia.org/r/1243855

gerritbot added a comment.Wed, Feb 25, 3:18 PM

Change #1243856 had a related patch set uploaded (by Urbanecm; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@wmf/1.46.0-wmf.16] SiteNoticeGenerator: stop adding per-variant classes

https://gerrit.wikimedia.org/r/1243856

gerritbot added a comment.Wed, Feb 25, 3:33 PM

Change #1243868 had a related patch set uploaded (by Urbanecm; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@wmf/1.46.0-wmf.16] LevelingUpManager: stop supporting multiple delay specifications

https://gerrit.wikimedia.org/r/1243868

gerritbot added a comment.Wed, Feb 25, 3:41 PM

Change #1243855 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.46.0-wmf.16] ExperimentManager: remove geForceVariant flag handling

https://gerrit.wikimedia.org/r/1243855

gerritbot added a comment.Wed, Feb 25, 3:41 PM

Change #1243856 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.46.0-wmf.16] SiteNoticeGenerator: stop adding per-variant classes

https://gerrit.wikimedia.org/r/1243856

gerritbot added a comment.Wed, Feb 25, 3:53 PM

Change #1243868 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.46.0-wmf.16] LevelingUpManager: stop supporting multiple delay specifications

https://gerrit.wikimedia.org/r/1243868

Stashbot added a comment.Wed, Feb 25, 3:54 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-25T15:54:42Z] <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1243855|ExperimentManager: remove geForceVariant flag handling (T416894)]], [[gerrit:1243856|SiteNoticeGenerator: stop adding per-variant classes (T416894)]], [[gerrit:rMW1243868d16f5|LevelingUpManager: stop supporting multiple delay specifications (T416894)]], [[gerrit:1243854|tests: Introduce MentorRemoverTest]]

Stashbot added a comment.Wed, Feb 25, 3:57 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-25T15:57:02Z] <urbanecm@deploy2002> urbanecm: Backport for [[gerrit:1243855|ExperimentManager: remove geForceVariant flag handling (T416894)]], [[gerrit:1243856|SiteNoticeGenerator: stop adding per-variant classes (T416894)]], [[gerrit:rMW1243868d16f5|LevelingUpManager: stop supporting multiple delay specifications (T416894)]], [[gerrit:1243854|tests: Introduce MentorRemoverTest]] synced to the testservers (see https://wikitech.wikimedia.or

ReleaseTaggerBot edited projects, added MW-1.46-notes (1.46.0-wmf.16; 2026-02-17); removed MW-1.46-notes (1.46.0-wmf.17; 2026-02-24).Wed, Feb 25, 4:00 PM
Stashbot added a comment.Wed, Feb 25, 4:01 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-25T16:01:24Z] <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1243855|ExperimentManager: remove geForceVariant flag handling (T416894)]], [[gerrit:1243856|SiteNoticeGenerator: stop adding per-variant classes (T416894)]], [[gerrit:rMW1243868d16f5|LevelingUpManager: stop supporting multiple delay specifications (T416894)]], [[gerrit:1243854|tests: Introduce MentorRemoverTest]] (duration: 06m 42s)

Sgs updated the task description. (Show Details)Fri, Feb 27, 5:01 PM
Sgs moved this task from Doing to Code Review on the Growth-Team (FY2025-26 Q3 Sprint 4) board.Fri, Feb 27, 5:03 PM
gerritbot added a comment.Fri, Feb 27, 5:44 PM

Change #1239893 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Experiments: dismantle ExperimentUserManager

https://gerrit.wikimedia.org/r/1239893

ReleaseTaggerBot edited projects, added MW-1.46-notes (1.46.0-wmf.18; 2026-03-03); removed MW-1.46-notes (1.46.0-wmf.16; 2026-02-17).Fri, Feb 27, 6:00 PM
gerritbot added a comment.Mon, Mar 2, 3:11 PM

Change #1240693 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Experiments: prefer FeatureManager over ExprimentManager

https://gerrit.wikimedia.org/r/1240693

Maintenance_bot removed a project: Patch-For-Review.Mon, Mar 2, 3:31 PM
Michael subscribed.Mon, Mar 2, 4:09 PM
Sgs mentioned this in T418764: Create a GrowthExperiments specific guideline for migrating existing instruments that use EventLogging to use TestKitchen.Mon, Mar 2, 4:31 PM
Sgs moved this task from Code Review to QA on the Growth-Team (FY2025-26 Q3 Sprint 4) board.Tue, Mar 3, 12:18 PM
Sgs updated the task description. (Show Details)Tue, Mar 3, 5:06 PM
DMburugu subscribed.Tue, Mar 10, 3:03 PM

The criteria for this task has been checked as met and the patches have been merged without any issues. This looks okay to resolve unless there's something I'm missing

Sgs closed this task as Resolved.Wed, Mar 11, 11:09 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits