Normal prio is fine, we should just be able to give an answer (with our hands waving a bit) when asked "what's needed to do this?" Timeline to have a hand-wavy answer: end of quarter-ish? I foresee tons of people asking for this in the next year.

The multiversion router could certainly be configured to key off of something other than the host: header. It would not take very much effort to implement, although the code in multiversion may contain live bobcats, or perhaps it's more probably than maybe.

Some possible criteria for routing a given request to a given branch:

• a hash of the client IP address.
• a cookie which routes users to the unstable branch, we could direct volunteer testers to a unique url that sets the cookie, and ask them be on the lookout for problems.
• We could (and probably should) send all wikimedia staff to the unstable branch when they are logged in.

There are some problems with sending random requests to the unstable branch, especially anonymous request, do to our aggressive cache layer. If we cache a broken page then it will potentially leak beyond our 'small sample group,' and additionally, the brokenness could persist in cache for quite a long time.

On IRC, @BBlack suggested that we might consider lowering our cache TTL to something like 2 weeks instead of 30 days, coinciding with changes to the release process. I'm sure there are other things we could do to mitigate the problems but I don't know of any brilliant solutions right now.

In T104398#1456131, @mmodell wrote:

On IRC, @BBlack suggested that we might consider lowering our cache TTL to something like 2 weeks instead of 30 days, coinciding with changes to the release process. I'm sure there are other things we could do to mitigate the problems but I don't know of any brilliant solutions right now.

I suspect it's possible we can drop our maximum TTL without a huge loss in cache hitrate, but that's just random speculation without data. It's entirely possible that doing so would cause an unacceptable plunge in hitrate. The whole reason that I brought that up was in the context of the idea of serial interoperability as a way to solve several related problems with deployments.

The general idea is that you'd pick a count of serial releases as a rolling interoperability window. For N=5, no change in a new release can break basic compatibility with any of the previous 4. Another way to think of that: it should be perfectly ok to run the cluster with all 5 of the most recent releases at 20% each. We might not want to do that because it would be confusing for users to see 5 versions of a feature, but fundamentally nothing would be broken for the site or the users. When there's a desire to make a breaking change (for example, deploying new javascript code to browsers which only works with matching new server-side code), one has to plan the phase-in of that feature such that the serial compatibility window is maintained.

Even if we only strictly forced this rule for N=2, it greatly improves the reliability of the deployment process at the appserver level (fast cluster-wide deployment to paper over incompatibilities no longer matters...), and our ability to cleanly roll backwards when necessary as well. But if we extend the window a bit, we can solve similar cache-level problems with it as well.

I'm oversimplifying by ignoring the dimension of the problem where we already run 3 releases for different wikis here, but: for instance, if we deploy 3 new versions every week, and we maintain serial compatibility for a window of 6 releases, then there can never be a compatibility issue caused by cached content with a lifetime of 2 weeks in our caches. That's the sort of context in which I think it's worth investigating the impact of cache lifetime reduction. If reducing from 30d to 14d only causes a small hitrate loss, but lowers the serial compatibility window from "unreasonable" to "practical", it might be worth it.

@BBlack: I think we should do essentially what you suggest, but instead of creating new branches each week we just have 2 or 3 long-lived branches and merge changes into them sequentially, e.g.

• Monday:
  • we accept proposed changes into the release tree (and only accept changes that are intended to maintain the compatibility window, as you described above).

git co wmf/group0
git merge $commit
git merge $commit
# (or git merge master, if we wanna be crazy)
# then... run thorough tests monday night.

• Tuesday:
  • If everything looks good, we create a new tag and push it to some percentage of users.
  • The groups can be defined either by using the current host-based grouping (group0.dblist, group1.dblist, etc) or some new groupings based on client IP, rand(), ...

# on tuesday, 
# 
git co wmf/group0
TAG=wmf/group0/{date or sequence number}
git tag -a $TAG
sync $TAG

• Wednesday:
  • Assuming things still look stable, we accept all the commits from group0 into group1
  • At this point we can start merging more changes into group0 (SWAT 2.0?)

git co wmf/group1
git merge group0
TAG=wmf/group1/{date or sequence number}
git tag -a $TAG
sync $TAG

Repeat up to as many parallel branches as we want to maintain, I think 2 would be a good start, maybe 3.

https://etherpad.wikimedia.org/p/rolling-deployments

I think the main points from a recent discussion were: in order to deploy to a percentage of traffic we need to do so quickly. In order to be able to move quickly we need to spot subtle cache-poisoning problems before they enter into a deployment pipeline.