The Mondays and Tuesday morning (SF-time) are in my opinion quite "hot" in terms of Code-Review stress.
Unlike any other day, code merged into master during this 24-hour period will go live in production very soon without much time for developers to discover friction in their local environments and/or for QA to find issues on the Beta Cluster.
The current reality is that our automated systems and Code-Review process are not good enough to provide a high enough confidence for immediate production rollout from master. We need testing in Beta and allow some time for post-merge review by maintainers of specific areas subscribed to merges in Gerrit. This is in part due to the wide hand-out of CR+2 rights to mediawiki/* repositories and the monolithic codebase of MediaWiki core.
As such, I propose that we change the way we cut branches and deploy them. I'm not sure about the specifics, but the end goal of this particular task is to ensure all commits receive at least 2-3 days of exposure to other developer and Beta Cluster before they end up running in a production environment. Regardless of whether it is just mediawiki.org and test wikis. And even ignoring test wikis, something merged Monday evening/Tuesday morning still hits production wikis like Wiktionary and Commons on Wednesday, which is 1 - 1.5 day post-merge, not 3 days.
One initial idea:
- Cut branch on Thursday instead of Monday.
This means new features and changes merged on Friday and Monday don't have the heat of going live very soon without any exposure in Beta Cluster and to developers' local environment. The downside is that any bug fixes we merge between Thursday and Monday we have to remember to back port. This gets harder as we won't notice them lacking a fix in Beta as the fix will be live there (with Beta running master).
As a general habit however, I think we should be clearer about identifying bug fixes from enhancements and always back port them to the latest wmf branch, no matter what day it is. So perhaps that's not an issue.
Alternatively, we could consider having Beta Cluster always run on the wmf+1 branch (which will be a non-prod branch between Thursday and Tuesday, a semi-prod branch on Tuesday evening and Wednesday, and Thursday it'll be the first to receive the next branch).