Page MenuHomePhabricator

Use/adopt search cluster ES management cookbooks for logging ES too
Open, Needs TriagePublic

Description

We have spiceracks cookbooks to handle common ES management (e.g. rolling restart) which might have some search cluster specific bits, but should be easily adaptable to the logging ES cluster as well. cc @Gehel @RKemper

Event Timeline

Change 777421 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] spicerack: add logging clusters to elasticsearch config

https://gerrit.wikimedia.org/r/777421

Note: Once OpenSearch compatibility mode is turned off, the ES libraries that Spicerack uses will no longer work. Spicerack will need to either bring in the opensearch-py library or call the API directly.

Change 777421 merged by Herron:

[operations/puppet@production] spicerack: add logging clusters to elasticsearch config

https://gerrit.wikimedia.org/r/777421

Change 785895 had a related patch set uploaded (by Herron; author: Herron):

[operations/cookbooks@master] sre.elasticsearch add logging clusters

https://gerrit.wikimedia.org/r/785895

Change 785895 merged by jenkins-bot:

[operations/cookbooks@master] sre.elasticsearch add logging clusters

https://gerrit.wikimedia.org/r/785895

ayounsi subscribed.

Note: Once OpenSearch compatibility mode is turned off, the ES libraries that Spicerack uses will no longer work. Spicerack will need to either bring in the opensearch-py library or call the API directly.

@colewhite my understanding is that this task has been resolved with @herron 's patch, and your comment should have its own distinct task (overall improvement to the cookbooks). Is that assessment correct?

@colewhite my understanding is that this task has been resolved with @herron 's patch, and your comment should have its own distinct task (overall improvement to the cookbooks). Is that assessment correct?

That's fair. Compatibility mode is turned off right now so it's worth testing to see if they work.

FYI, I have been working on writing a rolling restart/reboot cookbook for opensearch clusters as part of https://phabricator.wikimedia.org/T344798. I made sure to make it work with the logstash clusters over in eqiad and codfw. I took the opportunity to also define the logstash-eqiad and logstash-codfw cumin aliases, as the logstash aliases resolved to hosts in both datacenters.

The cookbook is sre.opensearch.roll-restart-reboot.