Context: The cross cluster settings are used to search across different projects, if they are invalid impact on users is that the crosslanguage & crossproject search is broken (silently, the widget does not show up).
The way to configure cross cluster seeds changed between elastic 5 and 6. Previously the key `search.remote` had to be set, now it must be `cluster.remote`.
It appears that the cluster `chi@eqiad` still contains references `search.remote` in addition to `cluster.remote`:
```lang=json
{
"search": {
"remote": {
"omega": {
"seeds": [
"elastic1034.eqiad.wmnet:9500",
"elastic1040.eqiad.wmnet:9500",
"elastic1038.eqiad.wmnet:9500"
]
},
"psi": {
"seeds": [
"elastic1052.eqiad.wmnet:9700",
"elastic1048.eqiad.wmnet:9700",
"elastic1050.eqiad.wmnet:9700"
]
}
}
},
"cluster": {
"remote": {
"chi": {
"seeds": [
"elastic1054.eqiad.wmnet:9300",
"elastic1074.eqiad.wmnet:9300",
"elastic1081.eqiad.wmnet:9300"
]
},
"omega": {
"seeds": [
"elastic1068.eqiad.wmnet:9500",
"elastic1076.eqiad.wmnet:9500",
"elastic1057.eqiad.wmnet:9500"
]
},
"psi": {
"seeds": [
"elastic1073.eqiad.wmnet:9700",
"elastic1075.eqiad.wmnet:9700",
"elastic1083.eqiad.wmnet:9700"
]
}
},
```
Unfortunately there does not seem to be a way to delete these settings as neither of:
```lang=shell
curl -XPUT -H"Content-Type: application/json" localhost:6102/_cluster/settings -d'{"persistent": {"search.remote": null}}'
curl -XPUT -H"Content-Type: application/json" localhost:6102/_cluster/settings -d'{"persistent": {"search" :{"remote": null}}}'
```
work as they return `persistent setting [search.remote], not recognized`.
What works is:
```lang=shell
curl -XPUT -H"Content-Type: application/json" localhost:6102/_cluster/settings -d'{"persistent": {"search" :{"remote": {"omega": {"seeds": null}}}}}'
```
But it seems to hit some BC code and affects the `cluster.remote` section leaving the `search.remote` section intact.
Re-adding the seeds to `cluster.remote` with:
```lang=shel
curl -XPUT -H"Content-Type: application/json" localhost:6102/_cluster/settings -d'{"persistent": {"search" :{"remote": {"omega": {"seeds": ["elastic1068.eqiad.wmnet:9500", "elastic1076.eqiad.wmnet:9500", "elastic1057.eqiad.wmnet:9500"]}}}}}'
```
does seem to force elastic to use these seeds instead of the old ones. There seems to be some dependency on the order in which these sections are loaded such that the old (and un-deletable?) `search.remote` section can be used as the source for the seeds.
AC:
- understand what's going on ([[https://discuss.elastic.co/t/weird-behavior-with-cross-cluster-settings-search-remote-vs-cluster-remote-elastic-v6-5-4/301980|question upstream]])
- cleanup the `search.remote` section from the chi@eqiad cluster settings