Page MenuHomePhabricator

xLab: update validation for traffic per wiki for Experiments
Closed, ResolvedPublic5 Estimated Story Points

Description

Coming out of T396045: xLab: follow up tasks for traffic and variations section and recent design UI review discussions, place validation rules and updates in a separate ticket to track progress, priority.

Description

Update validation rules and associated UI copy for the User traffic per wiki field set:

Acceptance Criteria

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Remaining traffic validation rulesrepos/data-engineering/mpic!217sfaciadditional-traffic-validationmain
Added basic Traffic validationrepos/data-engineering/mpic!198sfacitraffic-validationmain
Customize query in GitLab

Event Timeline

Links to the initial design specs for "Allow for additional wikis and increase of sample rate for activated experiments" and "Enforce max traffic percentages (...)" have been added to the task description.

None of the specifications have been reviewed with help from an engineer, so I recommend pairing to do a quick run-through whenever we're ready to address this task.

🆕 The specs were updated to remove the 'All wiki' field from the 'User traffic and variations' module. See T396201: xLab: Remove default sample size field (aka All wikis) from experiment form

Hey @cjming 👋🏻 Should this task also cover the implementation of the max. 50 wikis validation (see docs)? Or is it a better idea to create a separate ticket?

Hey @cjming 👋🏻 Should this task also cover the implementation of the max. 50 wikis validation (see docs)? Or is it a better idea to create a separate ticket?

good catch @Sarai-WMF - added it as a line item to ACs - thanks!

Milimetric moved this task from READY TO GROOM to Backlog on the Test Kitchen board.
Milimetric set the point value for this task to 5.Jun 10 2025, 3:52 PM

re: 3rd AC - Prohibit destructive changes to traffic while experiment is active (i.e. no decrease of sample rates and no removal of wikis)

presumably then decrease/removal is allowable if an experiment owner first deactivates an experiment and then turns it on again?

re: 3rd AC - Prohibit destructive changes to traffic while experiment is active (i.e. no decrease of sample rates and no removal of wikis)

presumably then decrease/removal is allowable if an experiment owner first deactivates an experiment and then turns it on again?

Waiting for confirmation from a tech mind (maybe @phuedx? 🙏🏻 ). But I would assume so since, in theory, enrollment would be reset if the experiment is activated again? I adjusted the copy of the info message to make this more obvious (i.e. mention "while active" instead of "once active"). I can adjust the content again in case I understood wrong:

image.png (132×1 px, 23 KB)

Waiting for confirmation from a tech mind (maybe @phuedx? 🙏🏻 ). But I would assume so since, in theory, enrollment would be reset if the experiment is activated again? I adjusted the copy of the info message to make this more obvious (i.e. mention "while active" instead of "once active"). I can adjust the content again in case I understood wrong:

image.png (132×1 px, 23 KB)

Can I suggest we tweak the wording slightly to

While an experiment is active, you can only increase wiki traffic percentages. Reducing traffic isn't allowed to avoid excluding users who are already enrolled in the experiment.

?

Unfortunately, enrollment wouldn't reset in any strict sense. The visitor would still have the same Edge Unique ID and the machine-readable name of the experiment wouldn't have changed.

Imagine an experiment with a treatment group that maps to [50000,50999]. The experiment owner then reduces the traffic size for the treatment group and it now maps to [50000,50499]. Those users who were assigned a bin in the range [50000,50499] will be in the second phase of the experiment.

If we do allow the experiment owner to reduce the traffic size, then we should make sure that the start date is updated so that the automated analysis doesn't include data from visitors who may be excluded from the experiment now.

Waiting for confirmation from a tech mind (maybe @phuedx? 🙏🏻 ). But I would assume so since, in theory, enrollment would be reset if the experiment is activated again? I adjusted the copy of the info message to make this more obvious (i.e. mention "while active" instead of "once active"). I can adjust the content again in case I understood wrong:

image.png (132×1 px, 23 KB)

Can I suggest we tweak the wording slightly to

While an experiment is active, you can only increase wiki traffic percentages. Reducing traffic isn't allowed to avoid excluding users who are already enrolled in the experiment.

?

Message updated. Thank you!

If we do allow the experiment owner to reduce the traffic size, then we should make sure that the start date is updated so that the automated analysis doesn't include data from visitors who may be excluded from the experiment now.

I think that then the exclusion issues should be prevented by default: If a user can deactivate an experiment, that means that its start date is in the past. This means that they'll be nudged by xLab to provide a future start date when they try to save a new configuration.

Per recent discussion, we will remove the 2nd + 3rd checkbox ACs in the description related to increase/decrease of traffic values from this task to T396650: xLab: enable changes in traffic allocation during active experiments

Whether we allow for decrease/destructive traffic changes while an experiment is not active should probably live in another ticket - spun one up just in case T396661: xLab: allow changes in traffic size for deactivated experiments to be re-run << @JVanderhoop-WMF feel free to decline this ticket if it doesn't make sense given our future plans. It does raise an interesting question tho of whether this could be another workaround for phased rollouts until said future plans materialize.

Sfaci renamed this task from xLab: update validation for traffic per wiki to xLab: update validation for traffic per wiki for Experiments.Jun 16 2025, 9:20 AM
Sfaci updated the task description. (Show Details)

Hey there, @Sfaci and @cjming! I'd say this is the most substantial pending task under SDS2.4.13. The hypothesis is off track, and I’m currently trying to assess how much time we’ll need to wrap it up. Could you share your thoughts on the current status and, if possible, provide an estimate of the time required to complete it? Thank you so much!

Note that, regarding this ticket, the following MR meets only, for now, the first AC (basic validation for Wikis and Traffic/Sample Rate).

Traffic validation has required changes in the way we were transforming sample_rate before sending that data to the backend. Now, because validation must run same way in both client and server sides, this transformation has been delayed/moved to be run directly in the backend just before saving that data in the database. That way we can use the validation suite for both inline client and server validation. There were a couple of tickets related to this so, the following MR address the mentioned before and also those tickets: T396201: xLab: Remove default sample size field (aka All wikis) from experiment form and T384107: Negative value sample size displays no validation and creates no experiment.

The pending ACs for this ticket are based on the changes mentioned above and everything will make more sense once we review and merge (if it happens) the mentioned MR

This task's parent Epic was resolved coinciding with the conclusion of hypothesis SDS2.4.13 at the end of FY24-25. This pending subtask can be considered foundational/essential work now, and until w decide whether it should be housed under a particular KR.

Change #1167322 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[operations/deployment-charts@master] xLab: Deploy v0.7.8 release to staging

https://gerrit.wikimedia.org/r/1167322

Change #1167322 merged by jenkins-bot:

[operations/deployment-charts@master] xLab: Deploy v0.7.8 release to staging

https://gerrit.wikimedia.org/r/1167322

Pending traffic validations of this ticket have been addressed in https://gitlab.wikimedia.org/repos/data-engineering/mpic/-/merge_requests/217 that is ready for review

Sfaci updated the task description. (Show Details)

Change #1175958 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] xLab: Deploy v0.8.1 release to staging

https://gerrit.wikimedia.org/r/1175958

Change #1175958 merged by jenkins-bot:

[operations/deployment-charts@master] xLab: Deploy v0.8.1 release to staging

https://gerrit.wikimedia.org/r/1175958

Change #1175961 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] xLab: Deploy v0.8.1 release to production

https://gerrit.wikimedia.org/r/1175961

Change #1175961 merged by jenkins-bot:

[operations/deployment-charts@master] xLab: Deploy v0.8.1 release to production

https://gerrit.wikimedia.org/r/1175961