Description
In principle, GrowthBook Integration + GrowthBook Customization scope's work only involves features that are currently present in Test Kitchen UI. We are focused on moving what we have currently in Test Kitchen UI to GrowthBook. But we haven't thought much about some possible new specific requirements in GrowthBook we might need to deal with.
The purpose of this ticket is to make a decision about what to do with some fields there are in the Analysis Settings section in GrowthBook:
- Are they required?
- Can a wrong value for these fields invalidate the experiment?
- Can they block or not, somehow, the minimal user flow we are focused on (registering and running experiments and seeing the corresponding results).
- Are they already being validated/checked as part of the default flow in GrowthBook UI? For example, before/when starting the experiment
Open questions
- There are three fields that represent Goal, Secondary and Guardrail Metrics. Are they required for GrowthBook experiments to work properly?
- They are included in the API response as settings.goals, settings.secondaryMetrics and settings.guardrails respectively
- Should we validate that Presto Analytics Iceberg is the selected Data Source? (In production there are two possible values for this field)
- It's id is included in the API response for any specific experiment as settings.datasourceId and its specific value can be fetched via /api/v1/data-sources endpoint
- Should we check that Bayesian is the chosen Statistic Engine? There are two possible values in the GrowthBook UI in the Metrics section
- It's included in the API body response as settings.statsEngine
- Is Decision Criteria a field with a default value in GrowthBook UI? Is there anything to do with it?
- Any other specific GrowthBook field to consider?
