Change Details

Our deployment plan for elastic 6 was: * Deploy split clusters with multi-cluster search * Deploy archive indices * Deploy elastic 6 It turns out that our current version of elastic has a [[ https://github.com/elastic/elasticsearch/issues/26833 | bug ]] that blocks deployment of multi-cluster search. The fix is shipped in elastic 5.6.3. We have a few options: == Deploy elastic 5.6.x before everything else Deploying elastic 5.6.x first will likely add 2 to 4 weeks to the deployment plan. This comes with the benefit of additional deprecation notices relevant to the elastic 6 upgrade == Built a custom elastic 5.5.2 with the bugfix The bug fix is a [[ https://github.com/javanna/elasticsearch/commit/fbee684bad52e2b7c3e51bf7cf25c5837da96f07#diff-24768980d79102206340502c94c52c35L18 | relatively small/isolated patch ]]. We could temporarily run a custom 5.5.2 with this bugfix. Seems iffy, we've never built custom elastic versions before. == Re-order deployment We can swap around the deployment order: * Merge the archive index patch, don't create new archive indices yet * Perform elastic 6 upgrade * Turn on cross-cluster search * Create archive indices This avoids the cross cluster search bug by keeping queries on the main cluster until we've upgraded to a fixed version of elastic.

Our deployment plan for elastic 6 was: * Deploy split clusters with multi-cluster search * Deploy archive indices * Deploy elastic 6 It turns out that our current version of elastic has a [[ https://github.com/elastic/elasticsearch/issues/26833 | bug ]] that blocks deployment of multi-cluster search. The fix is shipped in elastic 5.6.3. We have a few options: == ~~Deploy elastic 5.6.x before everything else~~ Deploying elastic 5.6.x first will likely add 2 to 4 weeks to the deployment plan. This comes with the benefit of additional deprecation notices relevant to the elastic 6 upgrade == ~~Built a custom elastic 5.5.2 with the bugfix~~ The bug fix is a [[ https://github.com/javanna/elasticsearch/commit/fbee684bad52e2b7c3e51bf7cf25c5837da96f07#diff-24768980d79102206340502c94c52c35L18 | relatively small/isolated patch ]]. We could temporarily run a custom 5.5.2 with this bugfix. Seems iffy, we've never built custom elastic versions before. == ~~Re-order deployment~~ We can swap around the deployment order: * Merge the archive index patch, don't create new archive indices yet * Perform elastic 6 upgrade * Turn on cross-cluster search * Create archive indices This avoids the cross cluster search bug by keeping queries on the main cluster until we've upgraded to a fixed version of elastic. == Hack around the bug * Force cross-cluster searches to query multiple indices to avoid the bug This lets us continue as is, with minor inconvenience.

Our deployment plan for elastic 6 was: * Deploy split clusters with multi-cluster search * Deploy archive indices * Deploy elastic 6 It turns out that our current version of elastic has a [[ https://github.com/elastic/elasticsearch/issues/26833 | bug ]] that blocks deployment of multi-cluster search. The fix is shipped in elastic 5.6.3. We have a few options: == ~~Deploy elastic 5.6.x before everything else~~ Deploying elastic 5.6.x first will likely add 2 to 4 weeks to the deployment plan. This comes with the benefit of additional deprecation notices relevant to the elastic 6 upgrade == ~~Built a custom elastic 5.5.2 with the bugfix~~ The bug fix is a [[ https://github.com/javanna/elasticsearch/commit/fbee684bad52e2b7c3e51bf7cf25c5837da96f07#diff-24768980d79102206340502c94c52c35L18 | relatively small/isolated patch ]]. We could temporarily run a custom 5.5.2 with this bugfix. Seems iffy, we've never built custom elastic versions before. == == ~~Re-order deployment~~ We can swap around the deployment order: * Merge the archive index patch, don't create new archive indices yet * Perform elastic 6 upgrade * Turn on cross-cluster search * Create archive indices This avoids the cross cluster search bug by keeping queries on the main cluster until we've upgraded to a fixed version of elastic. == Hack around the bug * Force cross-cluster searches to query multiple indices to avoid the bug This lets us continue as is, with minor inconvenience.