Page MenuHomePhabricator

Investigate open-source feature flagging/experimentation platforms
Open, Needs TriagePublicSpike

Description

TOINVESTIGATE

Acceptance

  • evaluation matrix, process and decision published on wiki
  • slide deck for leadership
  • presentation for leadership

Event Timeline

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptApr 27 2023, 9:08 AM

GrowthExperiments: https://phabricator.wikimedia.org/T335124#8795980

See also T288022: Support multidimensional user variants in GrowthExperiments and T323811: [EPIC] Community configuration 2.0: Factor Community configuration out of GrowthExperiments (especially the latter).

While I was walking through the GrowthExperiments codebase is that it doesn't bucket users deterministically, rather it rolls a dice and persists the result in the database (in the user_properties table). This approach won't scale to the size of our logged-out user audience.

There is also a PostHog - https://posthog.com/feature-flags - that btw is more a suite, but one of the features it provides is feature toggling.

A while back I put together a brief overview of how multi-armed bandit (something Ax supports) could be beneficial to our banner-based fundraising efforts (compared to traditional A/B testing): https://docs.google.com/presentation/d/14WGAIxMqN6TqdWk4l7Dg8jiZtn4rDyAU_ezbk1lKm_U/edit?usp=sharing

And Bayesian optimization (another thing that Ax supports) could help teams like Search Platform experiment with configurations.

Happy to chat about Ax and adaptive experimentation in general!