Background
FY24/25 SDS 2.4 covered the implementation of an initial Experimentation Lab, including:
- A UI to define and deploy experiments (the so-called xLab UI)
- A module that performs experiment enrollment at our edge and another in MediaWiki
- An SDK to get enrollment details for the user and sending experiment-related analytics back to our data lake
- Updates to EventGate to intake and decorate those events
- An experiment analytics pipeline
Experiment Platform intends to replace the UI to define and deploy experiments (1) and the experiment analytics pipeline (5) with GrowthBook.
During SDS 2.4, DPE SRE deployed GrowthBook and Experiment Platform did propose a way to connect GrowthBook to our data lake. However, there are there are still risks that we need to mitigate and questions that we need to answer:
- Risk: Querying our data lake will not be performant enough for our expected use of GrowthBook
- Risk: We can’t limit GrowthBook’s access to our data lake
Risk: GrowthBook’s integration with FerretDB is unstable- GrowthBook v4.1 officially supports FerretDB: https://blog.growthbook.io/growthbook-version-4-1/#ferretdb-support
- Unknown: We haven’t integrated GrowthBook with CAS-SSO. What does that look like?
- Unknown: We haven’t verified that the shape of the data that we’re collecting is compatible with GrowthBook
- Risk: Experiment Platform onboard WMF staff to GrowthBook to define and deploy experiments, only to find out that they can’t use it as they expected
Fortunately, because of both the architectural decisions we made during SDS 2 and GrowthBook’s modularity, we can replace them out of order, i.e. WMF staff can define and deploy experiments with xLab UI while analyzing them with GrowthBook’s experiment analytics pipeline. Moreover, we can run the current experiment analytics pipeline and GrowthBook’s in parallel.
Experiment Platform believes that GrowthBook should be integrated as an experiment analytics pipeline first because it’s non-disruptive and, per the above, that’s the greatest source of risk.
Proposed Subtasks
Deploy the latest version of GrowthBook backed by FerretDBSee T406578: Deploy Postgres for Growthbook and T406579: Deploy FerretDB for GrowthBook- Integrate GrowthBook with CAS-SSO
- Investigate connecting GrowthBook to our Data Lake via Presto and the Hive Metastore, including:
- Whether the connection is performant
- Whether the connection can be limited to a subset of tables