We discussed having the validator disable invalid experiments in GrowthBook, to help notify the experiment owner of the issue. Is this possible, and should it be included in the scope of this ticket if so?

We should have a definition of what valid means here. I suggest that a valid experiment MUST:

• Have a trackingKey and owner
• List between one and 100 wikis to run on
• Include a dateStarted
• Include one or more variations.

And it MAY:

• Include an end date. (If it does not, our conversion layer should add a default end date based on the start date. Eg: 7 days later.)

Questions:

• What requirements on variation names do we have and want to enforce?
• Do we want to check anything about the weighting of the variations?
• How do we want to support knowing whether an experiment is cache-splitting? This might be another piece the validator enforces having.
• Do we want to enforce the validation on experiments which are not started yet? I think not, since they may be drafts which don't include all the data.

In T421803#11786249, @KReid-WMF wrote:

We should have a definition of what valid means here. I suggest that a valid experiment MUST:

• Have a trackingKey and owner
• List between one and 100 wikis to run on
• Include a dateStarted
• Include one or more variations.

This sounds good to me - I will start with these initial requirements

And it MAY:

• Include an end date. (If it does not, our conversion layer should add a default end date based on the start date. Eg: 7 days later.)

I'm confused about end dates

in this view of the response schema, the "gets" (https://docs.growthbook.io/api/#tag/experiments/operation/listExperiments, https://docs.growthbook.io/api/#tag/experiments/operation/getExperiment) look like phases.dateEnded is required:

but in the "posts" (https://docs.growthbook.io/api/#tag/experiments/operation/postExperiment, https://docs.growthbook.io/api/#tag/experiments/operation/updateExperiment), only name and dateStarted of phases are required

I'll follow up in the shared channel to inquire about this more in depth

Questions:

• What requirements on variation names do we have and want to enforce?

will this be answered by {T422377: [Customize GrowthBook] Explore and evaluate what platform needs and what GrowthBook provides}?

• Do we want to check anything about the weighting of the variations?

I'm not sure what this means

• How do we want to support knowing whether an experiment is cache-splitting? This might be another piece the validator enforces having.

If we can do conditional custom fields in the GB UI, maybe we can add a field like in TKUI to distinguish between CS + NCS everyone experiments?
also maybe something to explore in the spike? {T422377: [Customize GrowthBook] Explore and evaluate what platform needs and what GrowthBook provides}

• Do we want to enforce the validation on experiments which are not started yet? I think not, since they may be drafts which don't include all the data.

This is interesting -- right now just by default, when I query for GB experiments, I see non-active experiments that have not yet started. Probably would be good to be explicit about what kind of experiments (active/inactive, started/not-yet-started) validation rules should apply to - again is this something that should be explored in the spike?

The spike on a valid config T422528: [GrowthBook Integration][SPIKE] Define a valid config went into many of these questions about what we want to enforce, and I'd defer to what we decided there.

Do we want to check anything about the weighting of the variations?

I'm not sure what this means

I was thinking about whether we want to enforce equally sized treatments - eg, prevent configurations from being control 10%, treatment A 30%, treatment B 60%. I'm not sure we do, because it would get in the way of doing multi-armed bandits. Having differently sized groups does complicate analysis though.

cjming opened https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/329

Implement GrowthBook validator and adapter

sfaci merged https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/329

Implement GrowthBook validator, adapter, and reconstructed experiment API endpoints

Change #1295092 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] test-kitchen: Update chart to add a new config property

https://gerrit.wikimedia.org/r/1295092

Change #1295094 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.3.8 release to staging

https://gerrit.wikimedia.org/r/1295094

Change #1295092 merged by jenkins-bot:

[operations/deployment-charts@master] test-kitchen: Update chart to add a new config property

https://gerrit.wikimedia.org/r/1295092

Change #1295094 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.3.8 release to staging

https://gerrit.wikimedia.org/r/1295094

Backend validation layer was added for GrowthBook experiments:

• validation/growthBookExperimentValidator.js - validateExperiment checks each experiment (trackingKey + owner presence, slug uniqueness against TK and GB experiments, deployment-day dates, 1–100 canonical wikis, variation/group allocation, stream name) and returns { valid, errors, wikis }.
  • validateAllExperiments runs this over the full GrowthBook response.
• util/experimentValidationRules.js - the pure rule functions are separate from the Vest UI form-validation suite.
• Invalid experiments are logged at error level with their joined rejection reasons and skipped, so they never reach adaptation/storage. The poll cycle continues rather than failing.
• Unit tests in test/unit/growthBookExperimentValidator.test.js and test/unit/experimentValidationRules.test.js cover missing/invalid fields, bad dates, and malformed variants/groups.

Change #1296660 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.3.9 release to production

https://gerrit.wikimedia.org/r/1296660

Change #1296660 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.3.9 release to production

https://gerrit.wikimedia.org/r/1296660