Looks like Elasticsearch 1.6.0 is about to be released! This is an exciting release for us because it provides a mechanism we can use for much much faster rolling restarts. This should be the last slow rolling restart!
Stakeholders: Cirrus engineers and operators
Benefits: 1/2 of faster deployments (36 hours drops to one hour, hopefully but we'll need to do some work on Cirrus too). With faster deployments we'll be ok deploying more frequently. There is another task we'll need (pausing updates) before we can have faster deploys.
Estimate: A week and a half. Maybe two and a half but probably not. We've done lots of Elasticsearch rolling upgrades.
So here is what we should do:
<s>1. Wait until Elasticsearch 1.6 goes live</s>
<s>2. Port all three of our plugins to it</s>
- Update vagrant to it
<s>4. Validate that everything still works in Cirrus with 1.6. All integration tests still pass.</s>
- Deploy new versions to beta.
- Let the whole thing soak for a week. I mean a week. Not 5 days. Not 2 days. New minor Elasticsearch releases need to spend a week or two in the wild before they hit our production cluster.
- Deploy to production.
So there are about 8 subtasks that we'll want to add to this task when we're ready to pick it up.