Page MenuHomePhabricator

Considering specific GrowthBook fields to be part of Validation/Customization
Closed, InvalidPublic

Description

Description

In principle, GrowthBook Integration + GrowthBook Customization scope's work only involves features that are currently present in Test Kitchen UI. We are focused on moving what we have currently in Test Kitchen UI to GrowthBook. But we haven't thought much about some possible new specific requirements in GrowthBook we might need to deal with.

The purpose of this ticket is to make a decision about what to do with some fields there are in the Analysis Settings section in GrowthBook:

  • Are they required?
  • Can a wrong value for these fields invalidate the experiment?
  • Can they block or not, somehow, the minimal user flow we are focused on (registering and running experiments and seeing the corresponding results).
  • Are they already being validated/checked as part of the default flow in GrowthBook UI? For example, before/when starting the experiment

Open questions

  • There are three fields that represent Goal, Secondary and Guardrail Metrics. Are they required for GrowthBook experiments to work properly?
    • They are included in the API response as settings.goals, settings.secondaryMetrics and settings.guardrails respectively
  • Should we validate that Presto Analytics Iceberg is the selected Data Source? (In production there are two possible values for this field)
    • It's id is included in the API response for any specific experiment as settings.datasourceId and its specific value can be fetched via /api/v1/data-sources endpoint
  • Should we check that Bayesian is the chosen Statistic Engine? There are two possible values in the GrowthBook UI in the Metrics section
    • It's included in the API body response as settings.statsEngine
  • Is Decision Criteria a field with a default value in GrowthBook UI? Is there anything to do with it?
  • Any other specific GrowthBook field to consider?

Event Timeline

Sfaci renamed this task from Consider specific GrowthBook fields to be part of Validation/Customization to Considering specific GrowthBook fields to be part of Validation/Customization.May 26 2026, 12:32 PM

There are three fields that represent Goal, Secondary and Guardrail Metrics. Are they required for GrowthBook experiments to work properly?

They are only required for analysis/results (including health checks). An experiment can be started without any metrics added.

Sometimes an experiment HAS to start without any metrics because:

  1. if the experiment targets metrics that haven't been defined in GB yet and
  2. if the metrics require new fact table(s) to be created

then it will be impossible to do that unless the experiment has started collecting data. When creating a new fact table, the query for the fact table MUST return some results to be able to save the fact table.

Should we validate that Presto Analytics Iceberg is the selected Data Source? (In production there are two possible values for this field)

In production the alternative is Presto Analytics Hive, which MAY be needed for some experiments (e.g. mobile apps until we get their data integrated into the Iceberg table)

Should we check that Bayesian is the chosen Statistic Engine? There are two possible values in the GrowthBook UI in the Metrics section

No, experiment owners are free to override that for their experiment. While we're keeping Bayesian as the default (https://wikitech.wikimedia.org/wiki/Test_Kitchen/Decision_Records/Keep_Bayesian_engine_as_default_in_GrowthBook) and recommend not switching, I don't think we should prevent someone from switching if they really want to. @JVanderhoop-WMF: do you concur?

Is Decision Criteria a field with a default value in GrowthBook UI? Is there anything to do with it?

Decision Criteria? I wonder if you mean the decision framework stuff. We've enabled it (per T426339#11946636) and I see it the in the UI:

Screenshot 2026-05-26 at 9.57.55 AM.png (1,902×778 px, 179 KB)

but not in the API output (decisionFrameworkSettings):

"settings": {
    "datasourceId": "ds_6v4betpmlupkb3v",
    "assignmentQueryId": "user_id",
    "experimentId": "we-1-8-account-creation-form-v1",
    "segmentId": "",
    "queryFilter": "",
    "inProgressConversions": "include",
    "attributionModel": "firstExposure",
    "statsEngine": "bayesian",
    "goals": [
      {
        "metricId": "fact__2423gnmnxlmpp7",
        "overrides": {}
      }
    ],
    "secondaryMetrics": [
      {
        "metricId": "fact__1hkkpmnp2f6vc",
        "overrides": {}
      },
      {
        "metricId": "fact__1hkkpmnp2hnjt",
        "overrides": {}
      },
      {
        "metricId": "fact__1hkkpmnp2gmmw",
        "overrides": {}
      }
    ],
    "guardrails": [],
    "regressionAdjustmentEnabled": false,
    "sequentialTestingEnabled": false,
    "sequentialTestingTuningParameter": 5000,
    "postStratificationEnabled": null,
    "decisionFrameworkSettings": {
      "decisionFrameworkMetricOverrides": []
    },
    "metricOverrides": []
}

Either way, that's not something to validate/require for experiments to be started.

There are three fields that represent Goal, Secondary and Guardrail Metrics. Are they required for GrowthBook experiments to work properly?

They are only required for analysis/results (including health checks). An experiment can be started without any metrics added.

Sometimes an experiment HAS to start without any metrics because:

. . .
. . .

Either way, that's not something to validate/require for experiments to be started.

Thank you very much @mpopov for this clarification. Now it's fully clear that this work is not needed. I'll close the ticket as invalid