Page MenuHomePhabricator

Fix cross-cluster seed settings in Search Platform production clusters
Closed, ResolvedPublic

Description

Per parent ticket and IRC conversation with @EBernhardson , we renamed the host prefix for our Search clusters from elastic to cirrussearch. This change is not reflected in in our cross-cluster seed settings. Creating this ticket to:

  • Ensure the correct hostnames appear in all 3 clusters (chi, psi, omega)
  • Confirm operation.

Event Timeline

bking changed the task status from Open to In Progress.May 1 2025, 4:47 PM
bking triaged this task as High priority.

Change #1140519 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: don't filter out self cluster settings

https://gerrit.wikimedia.org/r/1140519

Change #1140519 merged by Bking:

[operations/puppet@production] cirrussearch: don't filter out self cluster settings

https://gerrit.wikimedia.org/r/1140519

I've merged the above change and applied it to the hosts. After manually running sudo bash /usr/local/bin/set-cross-cluster-seeds_9xxx.sh (the same command run by the systemd timer), I can see the expected data in the OpenSearch cluster settings:

curl -H 'accept: Application/yaml' -s http://0:9200/_cluster/settings
persistent:
  cluster:
    remote:
      chi:
        seeds:
        - "elastic2093.codfw.wmnet:9300"
        - "cirrussearch2061.codfw.wmnet:9300"
        - "elastic2081.codfw.wmnet:9300"
        - "cirrussearch2081.codfw.wmnet:9300"
        - "elastic2084.codfw.wmnet:9300"
        - "cirrussearch2084.codfw.wmnet:9300"
        - "cirrussearch2111.codfw.wmnet:9300"
      omega:
        seeds:
        - "elastic2092.codfw.wmnet:9500"
        - "cirrussearch2092.codfw.wmnet:9500"
        - "elastic2100.codfw.wmnet:9500"
        - "cirrussearch2100.codfw.wmnet:9500"
        - "elastic2106.codfw.wmnet:9500"
        - "cirrussearch2106.codfw.wmnet:9500"
        - "elastic2073.codfw.wmnet:9500"
        - "cirrussearch2073.codfw.wmnet:9500"
        - "elastic2086.codfw.wmnet:9500"
        - "cirrussearch2086.codfw.wmnet:9500"
      psi:
        seeds:
        - "elastic2108.codfw.wmnet:9700"
        - "cirrussearch2108.codfw.wmnet:9700"
        - "elastic2076.codfw.wmnet:9700"
        - "cirrussearch2076.codfw.wmnet:9700"
        - "elastic2080.codfw.wmnet:9700"
        - "cirrussearch2080.codfw.wmnet:9700"
        - "elastic2083.codfw.wmnet:9700"
        - "cirrussearch2083.codfw.wmnet:9700"
        - "cirrussearch2115.codfw.wmnet:9700"

As such, I'm closing out this ticket. Feel free to re-open if things are not working as expected.

Change #1144639 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: don't filter out self cluster settings

https://gerrit.wikimedia.org/r/1144639

bking renamed this task from Fix cross-cluster seed settings in CODFW to Fix cross-cluster seed settings in Search Platform production clusters.May 12 2025, 6:59 PM

Change #1144639 abandoned by Bking:

[operations/puppet@production] elastic: don't filter out self cluster settings

Reason:

already addressed by Iaf4ff43fc982f52b9eef34dc6cdb66b3e3e74a07

https://gerrit.wikimedia.org/r/1144639