Just a couple of things to keep in mind related to recent changes made regarding how MediaWiki caches the Test Kitchen API response + some great ideas that Sam has shared recently:

• The TTL for the cache now is 10 minutes. It was 1 week at the time of the incident described in the ticket description. That would be the reason why we didn't see any change after updating the Test Kitchen API response (we only added some new fields). We thought that the cache wasn't being updated because we should have increased the cache version (from 1 to 2), we did it and we caused a new and a different issue. Now the cache for all wikis wouldn't be updated until the deployment train finishes on the following Thursday because the old code was running with a previous version of that cache (still 1)
• Any process that we described there as an outcome for this ticket won't be necessary when the change in the API response is backward compatible, because we won't need to update the cache version. And, because of the short TTL mentioned above, when the response is updated and is backwards compatible, we only need to wait 10 minutes to see how the cache is updated

In T422377#11974449, @cjming wrote:

this is great! i hope Custom Hooks are possible

In T422377#11973771, @KReid-WMF wrote:

Thank you for looking into this and writing it up!

For non-cache splitting experiments, have we decided how we will represent the different target maximums based on how common the action is (Reading / Editing / Account creation)? And if so, will we need customization to handle it?

Would we need to have all the current features working in GrowthBook?

GrowthBook API has been registered already as a service in the mesh (staging + production)

In T422943#11966043, @cjming wrote:

Adding reference here for Related OKRs and Owner/Team validation - https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/329#note_210225 discussion so we don't forget.

The main question is how to validate them as tags?

In the meantime, and via https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1295094, test-kitchen-next will be configured to consume the discovery.wmnet URLs mentioned above. That will allow us to start testing some new features related to T427025: [GrowthBook Integration] Enable Test Kitchen to consume the GrowthBook API on staging while we keep working on the remaining parts.

In T427278#11955357, @mpopov wrote:

There are three fields that represent Goal, Secondary and Guardrail Metrics. Are they required for GrowthBook experiments to work properly?

They are only required for analysis/results (including health checks). An experiment can be started without any metrics added.

Sometimes an experiment HAS to start without any metrics because:

. . .
. . .

Either way, that's not something to validate/require for experiments to be started.

In T415246#11955644, @phuedx wrote:

The issue was fixed by @hashar in https://gerrit.wikimedia.org/r/c/integration/quibble/+/1293735

Balthazar has confirmed that, in the meantime if needed, we could use the internal discovery domain: https://growthbook-next.discovery.wmnet:30443 and https://growthbook.discovery.wmnet:30443 for staging and production respectively

In T426110#11944982, @cjming wrote:

I think this is done? prod still needs the status filter update but afaict the 400 error has to do with this rather than with the api key

In T426687#11941166, @medelius wrote:

Yes, that's the one, @Sfaci ! I've updated the description to reflect the correct original start date.