Do we have a working list of what the platform needs? We will need valid config T422528: [GrowthBook Integration][SPIKE] Define a valid config, and ideally we want to enforce this in the GBUI and force privacy reminders around obligations and risk levels. (e.g. we implemented a way to show when configuration leads to medium- and high-risk data collection activities in TKUI. How might we do the same for GB?)

In T422377#11867051, @JVanderhoop-WMF wrote:

Do we have a working list of what the platform needs? We will need valid config T422528: [GrowthBook Integration][SPIKE] Define a valid config, and ideally we want to enforce this in the GBUI and force privacy reminders around obligations and risk levels. (e.g. we implemented a way to show when configuration leads to medium- and high-risk data collection activities in TKUI. How might we do the same for GB?)

This spike is precisely to explore that, which other features could be interesting, the needs of the platforms and so on. The description is still a TODO (because I'm still working on these tickets) but the spike goals are already listed above and, for now, would be:

• Define a list of what the platform needs
• Define a list of which customization ways GrowthBook provides
  • Request for features to GrowthBook are included in this step
• Map the needs and the available customization ways

Not sure if we will be able to enforce via GB UI all the validation we are going to need but I agree we should try to include all you have mentioned and all we can regarding T422528: [GrowthBook Integration][SPIKE] Define a valid config. I'm not even sure we can enforce the same validation rules we have right now in Test Kitchen UI. I think that's why we also have T421803: [GrowthBook Integration] Implement validation for GrowthBook experiment configurations where some logic will be implemented in our backend to check which GB experiments are valid to be delivered via Test Kitchen API. Our current implementation, Test Kitchen UI, is as open as we want and GB has some features to extend its logic but it won't be that flexible.

Regarding the Open Questions:

Would we need to have all the current features working in GrowthBook?

Exploration has shown that not all features we have in Test Kitchen UI are needed. As shown below, some fields and validation rules are not needed because they are already unused (Schema Type, for example), GrowthBook will be providing some validation rules by default (Some fields like experiment name or machine-readable one are required by default) and some others don't make sense in the new scenario (the ones about Start and End dates, for example). All details about this will be explained as answers in the following Open Questions and also as needed work the result of this Spike.

Do we need to keep all the existing form fields we currently have in Test Kitchen UI?

We wouldn't need to keep Schema Type and End Date as fields when registering experiments in the GrowthBook UI. Schema Type is not actually even needed or used in Test Kitchen UI and End Date will be initially pre-calculated by the backend (as startDate + 90 days) after an experiment is fetched by the poller, and also when new phases are added by GrowthBook, so no specific field nor validation rule will be needed.

Are the current validation rules needed also in GrowthBook?

Not all validation rules will be needed in this new phase where GrowthBook is the tool where experiment owners register their experiments. Some are provided by default (for example, experiment name and machine-readable one are required by default in the GrowthBook UI) and some others will be removed (endDate won't be a field anymore in the UI and its corresponding inline validation rules will be also removed)

Some other custom validations we have in Test Kitchen UI to ensure that the amount of traffic is a valid number won't be needed anymore in GrowthBook UI. The user is assigning the amount of traffic via a Slide instead of a TextField so wrong numbers are not a possibility.

What could we validate via GrowthBook UI/logic? Is it enough?

We have found that there are two scenarios here, depending on Custom Hooks are or not enabled for Experiments:

• Custom Hooks would allow us to implement pretty complex custom validation at the GrowthBook UI layer. That would allow us to alert inline experiment owners when a validation rule is not met while registering their experiments
• If Custom Hooks are not enabled for Experiments, we could define a Pre-Launch Checklist as a way to provide soft validation rules in the GrowthBook UI. But we should also implement same validation rules in the backend as a way to ensure they are met before including a specific experiment in the Test Kitchen API responses. That way, when a validation rule is not met, the backend validator will be able to skip that experiment and alert the corresponding experiment owner via T427593: [GrowthBook Integration] [Spike] Add alerting and de-duplication for GrowthBook experiment validation failures so they can fix its configuration.

Which features can/should we request?

Exploration has shown that we would need Custom Hooks to be able to provide hard validation rules at the GrowthBook UI level. We are currently in conversations with GrowthBook to have it enabled for Experiments. No answer at the time of writing this.

And regarding the Spike Goals:

Define a list of what the platform needs & Define a list of which customization ways GrowthBook provides & Map the needs and the available customization ways

There are two areas of work here:

Missing fields that are needed to register experiments to the GrowthBook UI + Fields we don't need anymore

• We would have to add phabricator_task, contact email, event_stream, contextual_attributes, risk_level and security_review_url as Custom Fields and according to the following configuration:
  • phabricator_task is an optional Custom url Field
  • contact_email is a required Custom text Field
  • event_stream is a required Custom text Field with product_metrics.web_base as default value
  • contextual_attributes is an optional Custom multiselect Field with all contextual attributes as possible values
  • risk_level is a required Custom enum Field with four possible values:
    • Risk assessment pending
    • Tier 1: High Risk
    • Tier 2: Medium Risk
    • Tier 3: Low Risk
  • security_review_url is an optional Custom url Field
• All possible values for owner and okrs need to be added as Tags for experiment owners to be able to choose the corresponding ones when registering/updating an experiment
• Schema Type won't be a field anymore. It's not needed
• End Date won't be a field in the GrowthBook UI. As we have mentioned above, it will be initially pre-calculated by the backend as startDate + 90 days and GrowthBook will keep it updated by default when new phases are created.
• User identifier type will be defined as a GrowthBook targetting attribute with all possible values
• Target Wikis will be defined as a GrowthBook targetting attribute with all possible values

Validation rules

Some additional validation rules to the ones that GrowthBook provides are needed. Some others won't be needed anymore.
Also, depending on whether Custom Hooks are available for Experiments, there will be two possible scenarios that will affect the way the corresponding rules will be implemented:

• Needed validation rules:
  • Machine-readable name for experiments (trackingKey in GrowthBook UI) should follow the pattern that Varnish requires
  • At least, while Test Kitchen is merging both GB + TKUI experiments, machine-readable name must be unique among both tools
• Related OKRs must be present (all valid tags must be added manually to GB)
• Team must be present (all valid tags must be added manually to GB)
• Stream name must be validated to ensure its value is an existing stream (can be checked via Action API as we are doing right now for TKUI)
• wikis (condition for MediaWiki database code attribute targeting) must be validated to check:
  • any of and equals to are the allowed operators
  • at least one wiki has been added to the list
  • wikis added to the list are valid (we could check values against the canonical-data list as we are doing right now with TKUI
• trafficSplit must be validated to check:
  • At least one wiki has been included
  • No more than 100 wikis has been included
• Variation machine-readable name should be validated to make sure the experiment owner has defined a custom value (by default numbers like 0, 1, . . .are provided by GrowthBook)
• No specific validation rules is needed for Start Date
• No specific validation rule is needed for End Date
• Risk level must be validated to make sure its value is according to the combination of contextual attributes that the user has chosen
• Security and Legal Review must be validation to make sure its value is according to the Risk Level the user has chosen
• A specific backend validation must be added to make sure an experiment won't be considered by Test Kitchen API when, having started already, its Risk Level is not still pending (it's defined as Risk assessment pending)

How/Where validation rules could be implemented

• If Custom Hooks are enabled for Experiments, all validation rules above (except the specific one for the backend) could be implemented via that new feature (even if an API requests are required, for example to use Action API)
• If Custom Hooks are not enabled for Experiments, we could define a Pre-launch Checklist as the way to define soft validation rules in the GrowthBook UI. Same validation rules would be implemented in the backend in the growthBookExperimentValidator component

Thank you for looking into this and writing it up!

For non-cache splitting experiments, have we decided how we will represent the different target maximums based on how common the action is (Reading / Editing / Account creation)? And if so, will we need customization to handle it?

this is great! i hope Custom Hooks are possible

re: trafficSplit - i can't recall if this is captured anywhere (i can dig around) but iirc from a sync mtg (and how the MVP of the GB poller/client currently works - I'm sure testing/QA/UAT will reveal more around UI confusion), we decided to limit how experiment owners could enter attribute targeting -- that the permitted operators we would accept are is any of and is equal to for the attribute MediaWiki database code.

Is this something that Custom Hooks could handle? Or something we can change as a form option in the GB UI?

The "AND"s and "OR"s are also interesting - parsing for targetingCondition gets hairy the more permutations there are with say for example values added for Target by Saved Groups and values added in multiple ANDs/ORs in Target by Attributes

In T422377#11973771, @KReid-WMF wrote:

Thank you for looking into this and writing it up!

For non-cache splitting experiments, have we decided how we will represent the different target maximums based on how common the action is (Reading / Editing / Account creation)? And if so, will we need customization to handle it?

We decided to tweak manually the database, right? So, in this case I have guessed we should tweak the current instruments table where TKUI experiment reside or trafficSplit column in the new experiments_cache table for GB experiments.

In T422377#11974449, @cjming wrote:

this is great! i hope Custom Hooks are possible

I would say that, for now, we should wait for custom hooks as something available. We proposed having Custom Hooks available for experiments but there is no answer yet. It seems that we will have to start working with the assumption that they are not available (to avoid delaying the work)

re: trafficSplit - i can't recall if this is captured anywhere (i can dig around) but iirc from a sync mtg (and how the MVP of the GB poller/client currently works - I'm sure testing/QA/UAT will reveal more around UI confusion), we decided to limit how experiment owners could enter attribute targeting -- that the permitted operators we would accept are is any of and is equal to for the attribute MediaWiki database code.

Is this something that Custom Hooks could handle? Or something we can change as a form option in the GB UI?

I think we discussed and agreed on what you mentioned but it's not written anywhere.
I would say Custom Hooks could handle something here. They would be called when an experiment is about to be created or updated and the full object is received as the input so, in theory, you can check whatever you want there and raise an error if something is not according your expectations.

If custom hooks are not available, I guess this work will have to be done in the backend, along with the other validation rules.

Anyway, thanks for remembering this! I can capture what you mention (any of / equals to operators) to be more precise regarding the validations for selected wikis. I'll update my comment above with these details.