Page MenuHomePhabricator
Create Task
Maniphest T268779

Support cloudelastic in spicerack elasticsearch
Closed, ResolvedPublic
Actions

Description

The problem

As an operator, I want to be able to use our standard cirrus-elasticsearch cookbooks (rolling-restart, rolling-reboot, rolling-upgrade, etc) when performing operations on cloudelastic. Currently we only support eqiad, codfw, and relforge; as a result operations like restarts on cloudelastic are done manually (see https://phabricator.wikimedia.org/T268770 SAL logs for an example of the process of manually restarting cloudelastic)

The solution

Add support for cloudelastic. The logic should all be the same so it should just be a matter of adding to our ELASTICSEARCH_CLUSTERS: Dict[str, Dict[str, Dict[str, str]]] config, which currently lives in Spicerack (as an aside, it should eventually live in Puppet).

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
elasticsearch: spicerack now supports cloudelasticRyan Kemperoperations/cookbooksmaster+1 -1
elasticsearch-cluster: support for cloudelasticRyan Kemperoperations/software/spicerackmaster+5 -0
Customize query in gerrit

Related Objects

Event Timeline

RKemper created this task.Nov 25 2020, 6:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 25 2020, 6:31 PM
gerritbot added a comment.Nov 25 2020, 6:32 PM

Change 643532 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/software/spicerack@master] cirrus: add support for cloudelastic

https://gerrit.wikimedia.org/r/643532

gerritbot added a project: Patch-For-Review.Nov 25 2020, 6:32 PM
Aklapper removed a subscriber: Discovery-Search (Current work).Nov 26 2020, 8:29 AM

(Please add project tags as project tags instead of subscribers - thanks!)

RKemper moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.Nov 30 2020, 6:23 PM
RKemper added a comment.Nov 30 2020, 6:31 PM
In T268779#6650603, @Aklapper wrote:

(Please add project tags as project tags instead of subscribers - thanks!)

ack! Thanks for the heads up

gerritbot added a comment.Dec 1 2020, 7:53 PM

Change 643532 merged by Ryan Kemper:
[operations/software/spicerack@master] elasticsearch-cluster: support for cloudelastic

https://gerrit.wikimedia.org/r/643532

Maintenance_bot removed a project: Patch-For-Review.Dec 1 2020, 8:10 PM
RKemper moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.Jan 11 2021, 4:43 PM
gerritbot added a comment.Jan 13 2021, 6:51 AM

Change 655810 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/cookbooks@master] elasticsearch: spicerack now supports cloudelastic

https://gerrit.wikimedia.org/r/655810

gerritbot added a project: Patch-For-Review.Jan 13 2021, 6:51 AM
gerritbot added a comment.Jan 13 2021, 7:01 AM

Change 655810 merged by jenkins-bot:
[operations/cookbooks@master] elasticsearch: spicerack now supports cloudelastic

https://gerrit.wikimedia.org/r/655810

Stashbot added a comment.Jan 13 2021, 7:04 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-13T07:04:35Z] <ryankemper> T266492 T268779 T265699 Restarting cloudelastic to apply new readahead changes, this will also verify cloudelastic support works in our elasticsearch spicerack code. Only going one node at a time because cloudelastic elasticsearch indices only have 1 replica shard per index.

Stashbot mentioned this in T266492: Restart elasticsearch clusters to apply readahead changes.Jan 13 2021, 7:04 AM
Stashbot mentioned this in T265699: 40-elasticsearch-readahead udev rule failing for cloudelastic100[5,6].
Maintenance_bot removed a project: Patch-For-Review.Jan 13 2021, 7:10 AM
Stashbot added a comment.Jan 13 2021, 10:53 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-13T22:53:02Z] <ryankemper> T266492 T268779 T265699 Restarting cloudelastic to apply new readahead changes, this will also verify cloudelastic support works in our elasticsearch spicerack code. Only going one node at a time because cloudelastic elasticsearch indices only have 1 replica shard per index

Stashbot added a comment.Jan 13 2021, 10:53 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-13T22:53:09Z] <ryankemper> T266492 T268779 T265699 sudo -i cookbook sre.elasticsearch.rolling-restart cloudelastic "cloudelastic cluster restart" --task-id T266492 --nodes-per-run 1

Stashbot added a comment.Jan 14 2021, 12:00 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-14T00:00:32Z] <ryankemper> T266492 T268779 T265699 Rolling restart of cloudelastic was successful

RKemper moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Jan 15 2021, 7:33 AM

The new cloudelastic functionality works properly.

Gehel closed this task as Resolved.Jan 20 2021, 8:28 AM
RKemper mentioned this in rCCKBedcd2fa0a730: elasticsearch: spicerack now supports cloudelastic.Dec 14 2022, 3:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits