Page MenuHomePhabricator

Support cloudelastic in spicerack elasticsearch
Closed, ResolvedPublic

Description

The problem

As an operator, I want to be able to use our standard cirrus-elasticsearch cookbooks (rolling-restart, rolling-reboot, rolling-upgrade, etc) when performing operations on cloudelastic. Currently we only support eqiad, codfw, and relforge; as a result operations like restarts on cloudelastic are done manually (see https://phabricator.wikimedia.org/T268770 SAL logs for an example of the process of manually restarting cloudelastic)

The solution

Add support for cloudelastic. The logic should all be the same so it should just be a matter of adding to our ELASTICSEARCH_CLUSTERS: Dict[str, Dict[str, Dict[str, str]]] config, which currently lives in Spicerack (as an aside, it should eventually live in Puppet).

Event Timeline

Change 643532 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/software/spicerack@master] cirrus: add support for cloudelastic

https://gerrit.wikimedia.org/r/643532

(Please add project tags as project tags instead of subscribers - thanks!)

(Please add project tags as project tags instead of subscribers - thanks!)

ack! Thanks for the heads up

Change 643532 merged by Ryan Kemper:
[operations/software/spicerack@master] elasticsearch-cluster: support for cloudelastic

https://gerrit.wikimedia.org/r/643532

Change 655810 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/cookbooks@master] elasticsearch: spicerack now supports cloudelastic

https://gerrit.wikimedia.org/r/655810

Change 655810 merged by jenkins-bot:
[operations/cookbooks@master] elasticsearch: spicerack now supports cloudelastic

https://gerrit.wikimedia.org/r/655810

Mentioned in SAL (#wikimedia-operations) [2021-01-13T07:04:35Z] <ryankemper> T266492 T268779 T265699 Restarting cloudelastic to apply new readahead changes, this will also verify cloudelastic support works in our elasticsearch spicerack code. Only going one node at a time because cloudelastic elasticsearch indices only have 1 replica shard per index.

Mentioned in SAL (#wikimedia-operations) [2021-01-13T22:53:02Z] <ryankemper> T266492 T268779 T265699 Restarting cloudelastic to apply new readahead changes, this will also verify cloudelastic support works in our elasticsearch spicerack code. Only going one node at a time because cloudelastic elasticsearch indices only have 1 replica shard per index

Mentioned in SAL (#wikimedia-operations) [2021-01-13T22:53:09Z] <ryankemper> T266492 T268779 T265699 sudo -i cookbook sre.elasticsearch.rolling-restart cloudelastic "cloudelastic cluster restart" --task-id T266492 --nodes-per-run 1

Mentioned in SAL (#wikimedia-operations) [2021-01-14T00:00:32Z] <ryankemper> T266492 T268779 T265699 Rolling restart of cloudelastic was successful

The new cloudelastic functionality works properly.