We will soon switch to fully automated branch cuts for the weekly MediaWiki train. In order to do so we must decide on the timing of the automation. This is likely to affect the workflow of some teams, so I am requesting feedback and discussion to decide on the ideal time for everyone.
The cut will be made at 02:00 UTC on Tuesdays.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | dancy | T196515 Automate the Train | |||
| Resolved | thcipriani | T242446 Decide timing of the automatic mediawiki branch cut | |||
| Open | None | T223410 Split group1 so that Commons and Wikidata aren't in the general group1, but their own buckets |
Change 563505 had a related patch set uploaded (by 20after4; owner: 20after4):
[mediawiki/tools/release@master] Deprecate make-wmf-branch
Is this just the branch cutting, which usually happens on Tuesdays? Or does it also affect the time of the actual deployment?
This is just the branch cutting. The deployment windows themselves are staying put.
Currently, the branch gets cut at varying times depending on who's deploying the train that week: typically sometime between 10-20 UTC (a large window of time :))
I think having the branch be cut at a fixed time would be good in general, because it would make the process more predictable. It would be helpful if the branch cut happened late enough in the day that there's an opportunity for people to merge patches on Tuesday morning in time for the train, but I also recognize that you can't both cut it late enough that developers on the US west coast have a chance to merge patches (say 18:00 UTC) and also cut it early enough that a releng team member in Europe can run the train (which currently happens at 14:00 UTC, when it happens).
In order to be able to deploy the train from Europe, the cut time probably couldn't be any later than 12:00 UTC though.
My vague proposal (linked to T223410: Split group1 so that Commons and Wikidata aren't in the general group1, but their own buckets) is to cut at ~09:00 UTC on Tuesdays, and then deploy to group0 at 10:00 UTC on Tuesdays (so that we could deploy to Wikidata at ~13:00 UTC on Tuesdays, giving WMDE proper time to respond to issues arising).
That doesn't leave much time to spot regressions on test wikidata. (3 hours)
Ideally we would have a suite of browser tests (which we have) to run on test wikidata in that time slot (if we endup with that deployment schedule)
Would it be feasible to automatically cut the branch on Monday, and also automatically deploy it to testwiki on Monday? I.e., the pre-deployment-window steps on https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Before_the_deploy_window except the one-time configuration steps.
Applying security patches would be a little tricky, but I'm sure it'd be solvable.
I think automatic cuts on Monday could work (but the reason we didn't before is because there are ~9 holiday Mondays a year which would be disrupted by this). Auto-deploying would involve trusting an automated process to apply security patches, which I wouldn't.
I would +1 the issues with Monday holidays. For our team, we support anytime so long as it's documented. I would ask that it is added to some shared cal that teams across Tech can see and likely also Product.
+1 to that
I'd quite like to see code land on test on a Monday.
With the other discussions that have been on going Wikidata could perhaps then be Tuesday?
Wednesday and Thursday as normal?
We will definitely document this on the Deployment Calendar -- is this what you had in mind?
One thing we discussed at the EngProd offsite (suggested by @Krinkle IIRC), was cutting the branch on the previous Thursday. This would give us more flexibility to, say, move the new branch to testwiki and/or testwikidata before the weekend if we wanted to. In short, it would allow more time to surface issues.
We've left this task open for a month, solicited feedback in all the usual places, and got some good feedback. It seems like most folks are fine with some automated time that is announced well ahead of time and documented everywhere.
We're not ready to move forward with the automatic branch cut, but when we do, we'll aim for:
Thursday 12:00UTC
This should allow plenty of time for a European deployment on Thursday (if we go that route). Additionally, by cutting the branch on Thursday, we have more flexibility in terms of rollout plans (even if we decide to stick to the regular Tues-Thurs cadence).
This is not a change that will be happening for a bit, when we're ready to throw the big switch we'll announce in all the usual places. This plan is tentative and subject to change.
We promise we'll make a lot of noise and update a lot of documentation before making this change a reality.
Thank you all for your feedback <3 -- closing out this task for now.
OK. We're ready to make this change :)
To minimize disruption of this change, I want to move forward incrementally and setup branch cut on Tuesday 02:00UTC starting 2020-06-16.
There's still time to make changes if needed, hopefully I haven't overlooked any processes that rely on the branch being cut later (if only by 12 hours or so, usually).
Branch has been cut at 02:00UTC since 2020-06-16—no complaint so far. If there's reason to move this, we can talk in a new task; calling this resolved.