Description
Recently we have been working in a new approach to deal with experiment configuration. So far we are dealing with feature variants (every feature has a name and some allowed values) but we want to explore a new approach based on treatment groups (a control group and a variable number of treatment ones that the user can configure). According to this new way to configure experiments, experiment owners will assign only groups to a specific experiment in such a way the feature code only will need to know which is the assigned group for every user enrolled in that experiment. According to the assigned group, the feature code will run some specific experience that they relate to that group. There will be a fixed control group to represent the existing experience and several treatment ones to represent the new experiences that the Product team wants to explore.
xLab UI
| Default state: Users find the control group predefined in the variations module. The ‘+ Add a variation’ button is available for them to insert a variation. | Variation added: Adding a variation results in a split of the enrollment percentage between groups. Users can input the name, machine-readable name and description of their variations. In the first iteration, the ‘+ Add a variation’ button becomes disabled after use. | Deletion confirmation: If users try to delete a Variation which fields are populated with data, a confirmation modal is thrown to avoid involuntary loss of information/progress. | Test activation confirmation: Users won't be able to edit their treatment group's details once their experiment has been activated. We'll mention this in the activation confirmation dialog as indicated in the screenshot. |
 |  |  |  |
Find flow and behavior annotations in the design specifications ➡ Figma | Miro
The new "Variations" module will be included right after the "Project and sample size" field in the A/B test form.
Note that, for now, only the required control group and one additional treatment work are allowed per experiment, all that according to the following rules:
- The Variation name field is a human-friendly name that instrument owners can edit freely
- The Machine-readable name field can be also edited freely by instrument owners, but the system will apply some rules that won't allow some special characters and spaces. The idea is to apply same restrictions that we are applying to generate the machine-readable name for the experiment. This works this way because this value will be the one that Product developers will use in the feature code to identify the assigned group for the current user. We'll cover these validation rules in a future task.
- Both the Variation name and the Machine readable name fields are required, and should be validated on submit like other required fields. (Improved validation will be implemented when we address this at a global form level. See T384707)
- The Description field can be edited freely. It will contain the group description
- There will be always a control group that will represent the existing change as it's right now (it's a required group). The instrument owner will be able to add any other treatment group they consider to test different features via the feature code
- The percentage assigned to every group will be equitable (50% each group if there are two groups, 25% each group if there are four, . . .)
- Once the experiment is activated, changes to related groups fields (friendly and machine-readable names and description) won't be allowed
- xLab will throw a dialog in the case the user tries to remove a group which fields contain any value (the copy for this dialog is still pending)
xLab API
We will have to update also the response for the /api/v1/experiments to reflect these changes. We will have to remove the features field from the response and add a new one called groups that will be an array that will contain only the machine-readable name for each group.
We will take also the opportunity to remove some unneeded data we are including currently in the /api/v1/experiments endpoint response. We are including there all the fields that belong to any specific experiment but the Experiment Manager don't need all of them to be able to enroll an user in some experiment. We could also update the endpoint to serve only experiments that have been activated (status === 1)
The following would be how an experiment would look from the API perspective with these changes:
{
"slug": "test_text_experiment",
"utc_start_dt": "2025-03-08T10:08:53.000Z",
"utc_end_dt": "2025-03-18T10:08:53.000Z",
"sample_rate": {
"default": 1
},
"groups": [
"control",
"bold_text",
"italic_text"
]
}xLab data model
The data model will be updated to store all the information related to groups (Group name, machine-readatable name and description) and the features field will be removed.
Acceptance criteria
- xLab UI: Experiment configuration is based on variations (instead of features)
- The "Variants" section in the A/B test configuration form is removed
- Following the design specifications, a required control is configured automatically, and experiment owners can add one more treatment
- The confirmation dialog that is shown when an experiment is activated will be updated to say "Variations" instead of "feature variants" when explaining which fields will no longer be editable
- A confirmation dialog is shown when the user tries to delete a group which fields contain any values
- The xLab data model allows to store groups (and the three related fields) and features field has been removed
- The /api/v1/experiments endpoint response has been update to include groups as a new field. It will be only an array with the machine-readable group names. Group name and Description won't be included as a part of the response
- Right now, we are limiting the UI to showing two groups: control and treatment (but the API and data model should allow more)
- The /api/v1/experiments response has been updated to remove all the non-necessary fields there: id, name, description, creator, owner, purpose, created_at, updated_at, task, compliance_requirements, sample_unit, environments, security_legal_review, was_activated and email_address)
- The /api/v1/experiments response has been updated to serve only the experiments that have been activated (status === 1) and the status field has been removed from the response
- Response format proposal reviewed and incorporated
