Our deployment plan for elastic 6 was:
* Deploy split clusters with multi-cluster search
* Deploy archive indices
* Deploy elastic 6
It turns out that our current version of elastic has a bug that blocks deployment of multi-cluster search. The fix is shipped in elastic 5.6.3. We have a few options:
== Deploy elastic 5.6.x before everything else
Deploying elastic 5.6.x first will likely add 2 to 4 weeks to the deployment plan. This comes with the benefit of additional deprecation notices relevant to the elastic 6 upgrade
== Built a custom elastic 5.5.2 with the bugfix
The bug fix is a [[ https://github.com/javanna/elasticsearch/commit/fbee684bad52e2b7c3e51bf7cf25c5837da96f07#diff-24768980d79102206340502c94c52c35L18 | relatively small/isolated patch ]]. We could temporarily run a custom 5.5.2 with this bugfix. Seems iffy, we've never built custom elastic versions before.
== Re-order deployment
We can swap around the deployment order:
* Merge the archive index patch, don't create new archive indices yet
* Perform elastic 6 upgrade
* Turn on cross-cluster search
* Create archive indices
This avoids the cross cluster search bug by keeping queries on the main cluster until we've upgraded to a fixed version of elastic.