⚓ T335482 Investigate feature flagging/experimentation platforms

Subject	Repo	Branch	Lines +/-
DNM: Example Unleash integration	mediawiki/core	master	+720 -2
DNM: Example Statsig integration	mediawiki/core	master	+186 -1
DNM: Example Growthbook integration	mediawiki/core	master	+185 -3

Status	Subtype	Assigned	Task
Declined		None	T335034 [Goal] Build the Experiment Control Plane
Declined		None	T335050 [Goal] M1: Experiment Control Plane: Experiment management UI for experiments on logged in users
Invalid		• VirginiaPoundstone	T360646 [Sprint 11 GOAL] SDS 2.5.6 Define User and Technical Requirements for experimentation flagging engine and analyze 3rd party option against those requirements
Resolved	Spike	phuedx	T335482 Investigate feature flagging/experimentation platforms

phuedx created this task.Apr 27 2023, 9:08 AM

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptApr 27 2023, 9:08 AM

phuedx updated the task description. (Show Details)Apr 27 2023, 9:12 AM

GrowthExperiments: https://phabricator.wikimedia.org/T335124#8795980

While I was walking through the GrowthExperiments codebase is that it doesn't bucket users deterministically, rather it rolls a dice and persists the result in the database (in the user_properties table). This approach won't scale to the size of our logged-out user audience.

phuedx moved this task from Backlog to Experiment Control Plane on the Metrics Platform board.Apr 27 2023, 9:56 AM

There is also a PostHog - https://posthog.com/feature-flags - that btw is more a suite, but one of the features it provides is feature toggling.

KSarabia-WMF subscribed.Aug 16 2023, 4:24 PM

phuedx updated the task description. (Show Details)Sep 11 2023, 10:58 AM

A while back I put together a brief overview of how multi-armed bandit (something Ax supports) could be beneficial to our banner-based fundraising efforts (compared to traditional A/B testing): https://docs.google.com/presentation/d/14WGAIxMqN6TqdWk4l7Dg8jiZtn4rDyAU_ezbk1lKm_U/edit?usp=sharing

And Bayesian optimization (another thing that Ax supports) could help teams like Search Platform experiment with configurations.

Happy to chat about Ax and adaptive experimentation in general!

• VirginiaPoundstone updated the task description. (Show Details)Jan 5 2024, 6:00 PM

• VirginiaPoundstone mentioned this in T360646: [Sprint 11 GOAL] SDS 2.5.6 Define User and Technical Requirements for experimentation flagging engine and analyze 3rd party option against those requirements.Mar 21 2024, 3:25 PM

• WDoranWMF added a project: Test Kitchen (Data Products Sprint 11).Mar 21 2024, 3:31 PM

• VirginiaPoundstone added a parent task: T360646: [Sprint 11 GOAL] SDS 2.5.6 Define User and Technical Requirements for experimentation flagging engine and analyze 3rd party option against those requirements.Mar 21 2024, 3:55 PM

phuedx updated the task description. (Show Details)Mar 25 2024, 2:41 PM

Haven't used it but this looks interesting: https://github.com/Unleash/unleash?tab=readme-ov-file

• VirginiaPoundstone updated the task description. (Show Details)Mar 28 2024, 3:55 PM

• VirginiaPoundstone updated the task description. (Show Details)

phuedx updated the task description. (Show Details)Mar 28 2024, 5:08 PM

• VirginiaPoundstone triaged this task as High priority.Mar 29 2024, 4:31 PM

phuedx claimed this task.Apr 4 2024, 2:58 PM

phuedx moved this task from Sprint Backlog to In Process on the Test Kitchen (Data Products Sprint 11) board.

phuedx set the point value for this task to 8.

phuedx renamed this task from Investigate open-source feature flagging/experimentation platforms to Investigate feature flagging/experimentation platforms.Apr 4 2024, 3:06 PM

See https://docs.google.com/spreadsheets/d/1pfsRDBk9QUAH9IzICxwDh7hUCYM_obB6RxpIwEgYrA0/edit (commentable by WMF).

tl;dr

The initial sample comprised 15 open core and closed source feature flagging/experimentation platforms. Platforms were evaluated against many dimensions and 9 were rejected. Of the 6 that remain, 4 are open core and 2 are closed source.

Next steps

@phuedx: Publish the spreadsheet
@VirginiaPoundstone: Do an equivalent product evaluation
@phuedx: Deliver an example integration of at least one of the platforms in a local MediaWiki development environment