The proposed end goal might cause more SWAT deploys will be used instead, working against the original goal.

I am also cautious whether staying longer at Beta Cluster will help. Maybe there are ways to make catching of issues at Beta Cluster more efficient, so that everything that will be caught will be caught as soon as possible, say, in 24 hours.

I support cutting branch earlier before deployment, preferable at a regular, known time, so people can easily plan for it.

We can easily cut the next branch before we deploy it (say, on Friday for the upcoming Tuesday train start) but we'll need yet-another-Beta-Cluster to test it on since we'll want the current Beta Cluster to continue to get all new code updates from master as they happen (modulo 10 minutes).

In fact, this was a planned goal for the "staging" cluster: to be a place where we could eg: freeze code the day before a deploy, run all of our integration/browser tests there, and then have a good feel for what we're deploying.

In T118212#1814215, @greg wrote:

We can easily cut the next branch before we deploy it (say, on Friday for the upcoming Tuesday train start) but we'll need yet-another-Beta-Cluster to test it on since we'll want the current Beta Cluster to continue to get all new code updates from master as they happen (modulo 10 minutes).

In fact, this was a planned goal for the "staging" cluster: to be a place where we could eg: freeze code the day before a deploy, run all of our integration/browser tests there, and then have a good feel for what we're deploying.

Just an idea but, if we're still too far away from sharing enough puppet code with prod and beta (and as such, farther away from easily creating more clusters), we could re-use the existing beta cluster meanwhile.

Like with prod, we could use multiversion in beta too. Requests for *.beta.wmflabs.org and *.staging.wmflabs.org would route through the same apaches and pick a different MediaWiki version based on the hostname.

In T118212#2142087, @Krinkle wrote:

Like with prod, we could use multiversion in beta too. Requests for *.beta.wmflabs.org and *.staging.wmflabs.org would route through the same apaches and pick a different MediaWiki version based on the hostname.

Yeah, that's a possibility as well.

Another idea for the short-to-mid term could be to have Beta Cluster only run master Tuesday-Thursday, but branch+1 from Friday onwards. Thus essentially using Beta Cluster as a "group -1" to mature the branch through the Friday and Monday workdays.

This would have the benefit of both: 1) A sort of "feature freeze" for two workdays during which only bugs are backported, and 2) a way for QA to increase their confidence level about the upcoming branch and to know when issues haven't been backported because they'd see the issue on Beta Cluster – knowing that that state will go live as-is.

Today the only representative state that goes live is during the minutes leading up to the branch cut on Tuesday. And of course, it's about more than just quality assurance. It's also about performance and production security, especially with our growing development community and merge access. In my opinion our current set up is a recipe waiting to blow up with a perfect storm.

I'd like to see the cost-benefits balanced so we can make an informed short-term decision. I can probably carve out some time to help with implementation if needed as well.