Our deployment plan for elastic 6 was:
- Deploy split clusters with multi-cluster search
- Deploy archive indices
- Deploy elastic 6
It turns out that our current version of elastic has a bug that blocks deployment of multi-cluster search. The fix is shipped in elastic 5.6.3. We have a few options:
Deploy elastic 5.6.x before everything else
Deploying elastic 5.6.x first will likely add 2 to 4 weeks to the deployment plan. This comes with the benefit of additional deprecation notices relevant to the elastic 6 upgrade
Built a custom elastic 5.5.2 with the bugfix
The bug fix is a relatively small/isolated patch. We could temporarily run a custom 5.5.2 with this bugfix. Seems iffy, we've never built custom elastic versions before.
We can swap around the deployment order:
- Merge the archive index patch, don't create new archive indices yet
- Perform elastic 6 upgrade
- Turn on cross-cluster search
- Create archive indices
This avoids the cross cluster search bug by keeping queries on the main cluster until we've upgraded to a fixed version of elastic.
Hack around the bug
- Force cross-cluster searches to query multiple indices to avoid the bug
This lets us continue as is, with minor inconvenience.