Page MenuHomePhabricator

Update Elastic puppet code to filter LVS config based on cluster membership
Closed, ResolvedPublic

Description

Each Elastic host runs 2 instances of Elasticsearch:

  1. The main instance (chi, ports 9200/9243)
  2. One of two secondary clusters (omega, 9400/9443 or psi, 9600/9643)

Per Slack conversation with @Vgutierrez , our current Elastic Puppet code is creating LVS config for both secondary clusters on all Elastic hosts, regardless of their actual secondary cluster membership. This will set off false positive alerts during the IPIP migration, so we'll need to clean this up.

Creating this ticket to:

  • Identify/update Puppet code
  • Confirm operation with Traffic team

Event Timeline

I've had a quick look at this to see if it is easy, but I don't immediately see a way of doing it without incurring a relatively high maintenance cost of lots of individual YAML files in hieradata/hosts.

We would need to match all of the cluster host names mentioned in the various cirrus.yaml files. e.g. here.

The only other way I could think of doing it would be to split the current role::elasticsearch::cirrus into two discrete roles, one with the omega instance the other with the psi instance installed.

Oh, maybe we could use hieradata/regex.yaml but that is still a bit fiddly.

Many of us in Data Platform SRE and/or Search Platform have looked this over, and I think we're in agreement that @Vgutierrez 's original suggestion is the least bad way to move the migration forward.

"The only alternative I see right now is setting profile::lvs::realservers::pools values per instance in hiera, cause all of those share the same puppet role and puppet profiles"

regex.yaml (suggested by Ben and others) could work, but seems more error-prone than hostvars, so we will move forward with the host-specific hieradata approach.

Change #1130162 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: add test hieradata to help with LVS migration

https://gerrit.wikimedia.org/r/1130162

Change #1138400 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elasticsearch: filter LVS config based on cluster membership

https://gerrit.wikimedia.org/r/1138400

Gehel subscribed.

Let's finish the OpenSearch migration first (T388610)

Change #1138400 merged by Bking:

[operations/puppet@production] elasticsearch: filter LVS config based on cluster membership

https://gerrit.wikimedia.org/r/1138400

Change #1159459 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elasticsearch: filter LVS config based on cluster membership

https://gerrit.wikimedia.org/r/1159459

Change #1159459 merged by Bking:

[operations/puppet@production] elasticsearch: filter LVS config based on cluster membership

https://gerrit.wikimedia.org/r/1159459

Mentioned in SAL (#wikimedia-operations) [2025-06-16T15:17:27Z] <inflatador> bking@cumin2002:~$ sudo cumin A:lvs-low-traffic 'run-puppet-agent' T387569

Change #1159517 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: move soon-to-be-decommed hosts to insetup role

https://gerrit.wikimedia.org/r/1159517

Change #1159517 merged by Ryan Kemper:

[operations/puppet@production] cirrussearch: move soon-to-be-decommed hosts to insetup role

https://gerrit.wikimedia.org/r/1159517

bking changed the task status from Open to In Progress.Jun 20 2025, 1:29 PM

After merging the above changes, we believe our Puppet code is correctly filtering based on cluster membership. Moving to "Needs Review" so @Vgutierrez can verify as time permits.

No response, so I'm going to go ahead and close this one out. Feel free to reopen if we missed anything!

Change #1130162 abandoned by Bking:

[operations/puppet@production] elastic: add test hieradata to help with LVS migration

Reason:

no longer needed

https://gerrit.wikimedia.org/r/1130162