A couple of questions/comments:

• I would assume that the validation process described here will be running only when the experiment is a Draft, as way to make that process easier for experiment owners. Mainly because the Checklist is not shown anymore once an experiment is running
• Experiment owners, when starting an experiment, could miss any Checklist item because it's not a hard blocker. Anyway, I would assume that once the experiment is running, the poller can stop checking the Checklist. The critical items are being validated already by the regular validation process (ExperimentKey, Risk Level, Traffic, . . .) and the other non-critical items (Description, Owner Team, Related OKR and Variation Ids) are just metadata that could be updated without altering the experiment results

Per slack thread, it was decided that the validation around tags is as follows:

• Experiments must be tagged with either Essential Work or OKR Work to pass validation
• A suggested (not required) custom field for the OKR number/name will be optional to fill out exp.customFields.related_okrs which will be a text value type

As we agreed in https://phabricator.wikimedia.org/T428805#12036814:

• The poller should validate the variation ids, because that has to match the code for the experiment to work. If it doesn't, then there's a major problem with the set up. (right?)
• I think the "soft rule" around description that you propose is a good one: the poller should not validate on Description, but we should have it in the checklist. We shouldn't "draft" an experiment that doesn't contain a Description and is running already.

And per slack thread we agreed on considering Related OKR as a soft validation. The Checklist will contain it as an item but the validation process won't fail when missing. And the experiment won't be drafted because of that. And same way of working would apply for the tag where the type of work will be set (Essential Work or OKR Work) and for the tag via which the Owner Team is provided

cjming opened https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/360

Draft: Warn on missing OKR, team, and description instead of drafting experiments

sfaci opened https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/362

Draft: Some refactoring to try to reuse the needed validation rules for when validating drafts

sfaci merged https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/360

Update validation rules for optional fields, variation ids

sfaci closed https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/362

Draft: Some refactoring to try to reuse the needed validation rules for when validating drafts

sfaci opened https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/363

Draft: Experiment Draft Checklist is updated after running validation

Change #1305785 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] growthbook: Updated chart to add API_RATE_LIMIT_MAX env var

https://gerrit.wikimedia.org/r/1305785

Your experiment Another Sample Experiment failed validation and its status has been reverted to draft in GrowthBook.

Validation errors:

• risk_level is 'Risk assessment pending'; experiment cannot run until resolved
• variation ids must be changed from the default integers

Change #1305785 merged by jenkins-bot:

[operations/deployment-charts@master] growthbook: Updated chart to add API_RATE_LIMIT_MAX env var

https://gerrit.wikimedia.org/r/1305785

@mpopov While reviewing this, @cjming proposed to skip experiments tagged as external imported when auto-completing Pre-Launch Checklist items (I didn't realize that) and we agreed on making that change. Those experiments were already bypassing validation so it makes full sense to bypass also Checklist auto-completion, which is actually a consequence of a previous validation step.

Extracted from a related conversation in Slack:

You all probably remember the conversation we had some weeks ago about the rate limit that GrowthBook API has (60 requests per limit). It wasn't a real issue at that time. But now that we want to fill the Experiment Checklists
automatically, we are going to make more requests than before (when the poller only was fetching experiments by making, for now, 1 request per poll = 4 requests per minute).
And now we have to make a request every time we have to mark one or more items as complete for a single Experiment. If we had 15 experiments, that would mean 60 requests per minute, the current limit (the poller runs an iteration every 15 seconds by default and we need 1 request per experiment to update its Checklist). I'm assuming here that all experiments would have something in their Checklist to complete which is the worst case but, the more experiments we have the more likely that scenario will be
I would say we have a few options to consider:

Define like a secondary poller/interval less frequent only for CheckLists. Every minute or so? Auto-completing CheckLists sounds like less critical than serving experiments via Test Kitchen API. I think we don't need to do that every 15 seconds. Depending on the case, Test Kitchen would be checking things much faster than a human can fix them
At the time of that conversation, I had another one with Ben where he found there is a env var, API_RATE_LIMIT_MAX, that we could probably use to increase that limit
A combination of both ideas above, which right now sounds the best thing to do

I also checked whether the GrowthBook API has some endpoint to make like bulk requests to complete the Checklists of several experiments at once but it seems there is nothing like that. What we have is the endpoint via which we can complete all the items we need for a Checklist for a single experiment

Based on the above a couple of changes where done:

• We have defined an environment variable in GrowthBook k8s chart, API_RATE_LIMIT_MAX, to increase the default rate limit, which was 60 requests per minute, to 100 req/min
• A couple of config properties have been added to test-ktichen service:
  • drafts_validation: true | false: to enabled/disable the process that validates and completes Pre-Launch Checklist items for Experiment Drafts
  • drafts_validation_interval: x: to specifiy how often that validation/completion process will run (every x iterations of the poller)

cjming merged https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/363

Experiment Draft Checklist is updated after running validation

Change #1306757 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[operations/deployment-charts@master] Add Test Kitchen config for draft validation

https://gerrit.wikimedia.org/r/1306757

Change #1306759 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.6 release to staging

https://gerrit.wikimedia.org/r/1306759

Change #1306757 merged by jenkins-bot:

[operations/deployment-charts@master] Add Test Kitchen config for draft validation

https://gerrit.wikimedia.org/r/1306757

Change #1306759 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.6 release to staging

https://gerrit.wikimedia.org/r/1306759

Change #1310139 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploying v1.4.8 release to production

https://gerrit.wikimedia.org/r/1310139

Change #1310139 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploying v1.4.8 release to production

https://gerrit.wikimedia.org/r/1310139

Because of the issue we have been dealing with via T431015: GrowthBook staging sluggish and throws authentication errors, the Checklist autocompletion feature will be fully disabled for now. We have found that the number of requests done by that feature (about 20 req/minute) was creating a high number of concurrent jobs that were competing for same resources (to access and modify some structures) and that has been causing a excessive high CPU load in the GrowthBook's database.

For now Test Kitchen UI will be working without this feature, which means that the poller won't be autocompleting the Pre-Launch Checklist items. It's not a blocker, because users will be able to do it themselves.

Related, and hopefully the ultimate fix for this, it seems that Custom Hook for experiments is getting closer so that would mean that we would need that autocompletion feature because we would able to validate everything inline (invalid experiment couldn't be registered/updated).