Page MenuHomePhabricator

Experiments should always start at 14:30 UTC
Closed, ResolvedPublic3 Estimated Story Points

Description

Current behavior for GB experiments is an experiment started before 14:30 UTC waits until 14:30 UTC to start. An experiment started _after_ 14:30 UTC starts immediately, and has the time backdated to 14:30 the same day.
This is not ideal for a few reasons:

  • GB's UI shows information that is in fact not what's happening in the system
  • GB isn't respecting the notion of a deployment window, which is specifically intended to limit the time(s) that production behavior can change. Other teams are affected by these changes.

Acceptance Criteria:

  • Update the behavior to start during the window:
    • If start before 14:30 UTC, experiment should turn on at 14:30 UTC that day
    • If start between 14:30 and 15:00 UTC (i.e. the end of the deployment window), it can start immediately -- it's within the window
    • If started after 15:00 UTC, the experiment should turn on in the next deployment window and the start date is updated to reflect this.
  • The same behavior applies to experiments which are scheduled to start as well as those started immediately with the start button.
  • Update the item in the pre-launch checklist about start dates to include acknowledging that the experiment start time will be during the next valid deploy window

Event Timeline

JVanderhoop-WMF moved this task from Incoming to READY TO GROOM on the Test Kitchen board.
KReid-WMF updated the task description. (Show Details)
KReid-WMF set the point value for this task to 3.
KReid-WMF moved this task from READY TO GROOM to Backlog on the Test Kitchen board.

I think we have to consider a few things differently regarding the AC that says the following:

  • The same behaviour applies to experiments which are scheduled to start as well as those started immediately with the start button.

So far we weren't considering scheduled experiment for validation and any other processing and GrowthBook considers, internally, scheduled experiments as Draft until they really start. That means that their status will be draft while they are just scheduled. Once the scheduled date is reached, the experiment will start running automatically. Also, a new property is set when an experiment is scheduled, statusUpdateSchedule, that contains the date for when the experiment has been scheduled to start (startAt). But someone has to approve this. Once someone does it, a new property is set for that experiment, nextScheduledStatusUpdate that will contain two fields, type: 'start' and date:'YYYY-MM-DDTHH:mm:ss.000Z' which means that the experiment will start at that time. If nobody approves the scheduled date, the experiment won't run at the scheduled date as statusUpdateSchedule and it will continue being a Draft

Once the scheduled date has been reached and the experiment starts automatically, statusUpdateSchedule remains untouched, nextScheduleStatusUpdate is set to null and currentPhase.dateStarted is set to the scheduled date.

Because drafting validation is disabled due to some performance issues we saw and, anyway, that feature will be disappear pretty soon (once we implement T436106: [Epic] GrowthBook Custom Hooks), it seems that the best approach here would be to filter running experiments as experiment.status === 'running' || (experiment.statusUpdateSchedule && nextScheduleStatusUpdate), as a way to include all the running experiments along with those ones that are scheduled + approved already (now we are just filtering by status === running` to consider only the experiments that are effectively running). Then, for running experiments,phases.dateStarted will be adjusted to be aligned with the deployment window datetime and, for scheduled ones, dates in statusUpdateSchedule and nextScheduleStatusUpdate will be also adjusted same way. We can leave GrowthBook update the currentPhase.dateStarted once the experiment starts. It will do according to the scheduled dates we have updated already. All this will be possible because the function that adjusts/updates start dates is also updating the corresponding date in GrowthBook via its API.

A simpler approach would be just doing nothing regarding the scheduled dates I have mentioned above and waiting until any scheduled experiment starts running to leave the poller does its part by updating currentPhase.dateStarted to be aligned with the deployment window date as it does for any running experiment. Even if the experiment was wrongly scheduled (for a Friday, for example), it would never start at that time because its start date would be updated to start on the following deployment window. I'll follow this approach for now

As I have mentioned in Slack:

Once that both this task and T433733: Allow experiments in test environments to start on days outside of Mon-Thurs are merged, I have realized that now that start dates are adjusted to be always according to a deployment window (first ticket), the poller won't need to validate dates anymore (we were only validating there that start dates were according to deployment windows).
And now there is also a new config property, validate_deployment_windows, (second ticket) to allow developers start experiments at any time when working locally/on staging, so date validation is not needed in those cases either. > And in production, the date/normalizer adjuster will work to make all experiments start during a deployment window. That adjustment, when enabled, happens before the validation so there won't be anything to validate regarding start dates and deployment windows.
The above is not an issue itself because everything is working perfectly. It's only that we are running a validation process that validates dates that are always valid because they were adjusted in a previous step. I'll push a follow-up MR to remove the affected function and its related tests.

So this task is in "Needs Review" due to the above. The related MR is https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/386

Change #1337862 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.5.4 release to staging

https://gerrit.wikimedia.org/r/1337862

Change #1338098 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.5.4 release to production

https://gerrit.wikimedia.org/r/1338098

Change #1337862 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.5.4 release to staging

https://gerrit.wikimedia.org/r/1337862

Change #1338098 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.5.4 release to production

https://gerrit.wikimedia.org/r/1338098

Change #1339043 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.5.5. release to staging

https://gerrit.wikimedia.org/r/1339043

Change #1339043 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.5.5. release to staging

https://gerrit.wikimedia.org/r/1339043

Change #1341040 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.5.5 release to production

https://gerrit.wikimedia.org/r/1341040

Change #1341040 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.5.5 release to production

https://gerrit.wikimedia.org/r/1341040