FY25-26 SDS2.3.1 Field of Dreams
Product Requirements
STATUS: Not started
| Reviewer | Date approved |
|---|---|
| Julie van der Hoop | 2026-03-25 |
| Karen Hernandez | 2026-03-25 |
| Katherine Reid | 2026-03-26 |
| Sam Smith | 2026-03-25 |
| Clare Ming | 2026-03-26 |
Working PRD - FY25-26 SDS2.3.1 Field of Dreams - GrowthBook connection components
Objective/Hypothesis
If we validate and adapt GrowthBook's API to our own, we can use GrowthBook as the canonical experiment control plane, and if we regularly poll GrowthBook's API, we can deliver experiments configured in GrowthBook quickly and easily.
How does this objective/hypothesis relate to organizational goals?
This hypothesis supports the 2025-2026 fiscal year annual plan for product and technology department to deliver on Objective SDS2, KR2.3, which states:
SDS Objective:
Product managers can quickly, easily, and confidently evaluate the impacts of product feature changes in Wikipedia
Key Result:
By end of Q4, all new Test Kitchen experiments are configured in GrowthBook.
Why do this?
We want to enable teams to configure, launch, analyze and repeat experiments — including routine retention measurements — without requiring multi-person coordination or manual handoffs across systems. We’ve chosen GrowthBook to serve as the authoritative system for experiment configuration and analysis, with Test Kitchen UI handling instrumentation only.
Timeline
By the end of the 4th quarter of FY25-26, we would like to have the GrowthBook connection components built and tested as critical infrastructure for using GrowthBook as the definitive experiment control plane for WMF teams.
Risks
| Risk | Description | Status | Notes |
|---|---|---|---|
| GrowthBook gets bought out | GrowthBook is sold or bought and is no longer open core, or available for licensing per contract negotiations | Mitigating | Varnish and MediaWiki will not talk to GrowthBook. Instead, TKUI will adapt GrowthBook’s API responses to those needed by Varnish and MediaWiki. |
| GrowthBook can’t handle the load | Mitigating | Implement caching in TKUI | |
| Switching from TKUI to GrowthBook causes event loss and causes an experiment to fail | Mitigating | TKUI will adapt and merge experiments from GrowthBook initially and while TKUI experiments are read-only. We will disable TKUI experiment functionality when all TKUI experiments are inactive | |
| Capacity and timing | Even though we’ve estimated that the work will not be too extensive in time, we’re less than 90 working days away from the end of the FY. We have the SDS2 offsite in April and other work we’ll need to tackle as part of SDS2.4. This can lead to not meeting our deadlines on time and the work spilling over to FY26/27. | Open | |
Who is involved
| Overview - DACI | |||
|---|---|---|---|
| Driver | Approver | Contributors | Informed |
| Clare Ming | Julie van der Hoop, Sam Smith, Katherine Reid | Steering committee: Karen Hernandez 2.3 Additional contributors: Santiago Faci, Mikhail Popov | SRE, PA |
| Details about roles and activities | |||
|---|---|---|---|
| Team/Role | Type | Individuals | Sample Activities |
| Experiment Platform | Development team | Katherine Reid(EM), Julie van der Hoop(PM), Sam Smith(Tech Lead), Mikhail Popov(PA), Santiago Faci(IC), Clare Ming(IC) | Technical implementation, Architecture consultation |
Requirements
Hypothesis Requirements
The end product/result for the Experiment Platform team should be a canonical experiment control plane in GrowthBook that proves GrowthBook-configured experiments have their experiment config added to experiment events as expected.
Success Criteria
- Experiments configured in GrowthBook for everyone and logged-in use cases propagate their configuration correctly through to the Test Kitchen backend (API responses) and Test Kitchen extension’s reading/processing of said configuration:
- GrowthBook-configured experiments appear with the correct JSON data in the TK experiment API endpoints.
- Test Kitchen extension reads GrowthBook experiment configs accurately to populate experiment events correctly in both PHP and Javascript SDKs.
- Further proof will be captured in an ensuing, separate hypothesis for running A/A synthetic experiments via GrowthBook.
- Frontend modifications to TKUI include:
- Providing read-only views of GB-configured experiments
- Disabling launch/edit views of TKUI-configured experiments
Target Outcomes
The goal is to:
- Decouple Test Kitchen UI from experiment configuration/analysis and use GrowthBook as the definitive experiment control plane.
- Provide product teams with industry-standard (not bespoke), expedient and efficient experimentation capabilities with automated analytics driving more timely data-driven decision-making around feature development and adoption.
- Enable teams to expeditiously capture metrics
What is out of scope?
There are various elements that we will consider out-of-scope for validation of this hypothesis:
- Backend modifications to TKUI:
- removing experiment configuration code and clean up related to disabling creating/updating experiments i.e. experiment store (remove save, update), components, validation, tests
Background & existing research or documentation
- Miro << design sprint architecture diagrams
- SDS2: Experimentation Platform FY25/26 - https://docs.google.com/document/d/1E_fsuy5rjQGIUoOGXWoiVSk4GhZAXI7zN0lT0m9AAL0/edit?tab=t.2mc7zhvxb50k#heading=h.48hxghbn2xko
- Design Sprint Tracker - https://docs.google.com/document/d/1UJK9rn7X08q7tGEpp-a67TYZBUtKf1u1c1gIh45GTNY/edit?tab=t.ch58e8mq0co
Open questions
- The success metrics delineated above (and milestones outlined below) are quite broad in scope. In terms of defining “done” for this body of work, do we need a more granular list of requirements?
- See last Milestone: experiments configured GrowthBook and Test Kitchen UI are served to Varnish and MediaWiki
- See Success Criteria above
- Does this hypothesis include completely cleaving off TKUI from experiment configuration?
- No
- Is it just proving that the connecting components we build work?
- Yes
- Does having 100% of experiments completely on GB by end of Q4 mean we also do the necessary work to prevent experiment config on TKUI?
- Not in this hypothesis
- Are TKUI frontend modifications (removing experiment config launch/edit capabilities and providing read-only views of GB-configured experiments) in scope?
- Yes - in scope as in just disabling on the frontend (not removing backend code related to functionality)
Milestones
- Initial development of the GrowthBook connection components:
- Poller
- Fetches (caches?) GB API every N minutes
- Validator
- Disables misconfigured experiments
- Adapter
- tkui- prefixed experiments to distinguish between same-named experiments (edge case)
- namespacing
- Caching
- Stitcher (temporary)
- There will be an interim period wherein we will need to combine experiments configured in TKUI and GB for the experiments’ API endpoints.
- Logging
- This will likely be part of the observability/introspection hypothesis
- Some logging will be included in active development of the components but will need to be expanded/refined during observability implementation efforts.
- Poller
- Experiments configuration in GrowthBook and Test Kitchen UI are served to Varnish and MediaWiki (i.e. “Stitcher”)
Dependencies
None so far - we decided during the design sprint not to set up a separate service for the GrowthBook connection pieces but to build the components as part of the Test Kitchen nodejs application which we wholly own.