Page MenuHomePhabricator

Pull Elasticsearch config out of Spicerack
Closed, ResolvedPublic3 Estimated Story Points

Description

Any changes to spicerack require a manual deploy, which is painful. Move the Elasticsearch cluster config into puppet.

This is where the config that needs to be pulled out lives: https://gerrit.wikimedia.org/g/operations/software/spicerack/+/master/spicerack/elasticsearch_cluster.py#23

Todo:

  • Create a YAML spicerack configuration file that mimics the current config constant. Spicerack configurations are managed in puppet. This configuration can be static at first, but ideally it should be created from state already in puppet (I'm not sure if this is easily doable - @EBernhardson might have some ideas)
  • In Spicerack, replace the configuration constant with the same data, but loaded from a configuration file that is passed to the elasticsearch_cluster module. See other examples of load_yaml_config in the Spicerack class.

Event Timeline

Gehel set the point value for this task to 3.Mar 29 2021, 3:30 PM
Gehel updated the task description. (Show Details)
Gehel added a subscriber: EBernhardson.
Gehel updated the task description. (Show Details)

Change 716532 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/software/spicerack@master] elasticsearch: begin pull out spicerack config

https://gerrit.wikimedia.org/r/716532

Some context on the general plan from my convo with SDineshKumar (who is going to take a swing at this)

(1) Change spicerack so that it is still hardcoded, but instead of directly hardcoding the config in the python dictionary as it does currently, have it hardcode the contents of the config in a yaml file and parse that file
(At this point spicerack will work the same but now we've changed the internals so that it's reading a YAML file, except instead of reading an actual file we're "mocking" the file read by having the contents of that file hardcoded in a variable in the code)
(2) Deploy puppet change to actually write the file to disk on `cumin*` hosts (cumin hosts are where we run cookbooks from)
(At this point the puppet code is writing the file but spicerack isn't reading it, so it's effectively a no-op but we can verify that the file is where it should be)
(3) Change spicerack to read from the actual file on disk rather than mock it. This should be minimal code change because we've already refactored in step #1
So step #3 is the only point at which we could break something, but it should just work if we did steps #1 and #2 properly

Hi @RKemper

Due to family situations, I couldn't pick this up yet. I will pick this up by 31st Oct.

Thanks
Dinesh

Change 768816 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] elasticsearch: move cluster configuration to puppet

https://gerrit.wikimedia.org/r/768816

Change 768816 merged by Bking:

[operations/puppet@production] elasticsearch: move cluster configuration to puppet

https://gerrit.wikimedia.org/r/768816

Change 716532 merged by Volans:

[operations/software/spicerack@master] elasticsearch: load config from yaml

https://gerrit.wikimedia.org/r/716532

razzi added a subscriber: razzi.

This is done! Note that the relforge host has been updated to relforge1004

razzi@cumin1001:~$ sudo python3
>>> from spicerack import Spicerack
>>> s = Spicerack(verbose=True, dry_run=True)
>>> ec.__dict__
KeyboardInterrupt
>>> ec=s.elasticsearch_clusters("relforge", ("some_core_dc",))
>>> ec.__dict__['_clusters'][0].__dict__
{'_elasticsearch': <Elasticsearch([{'host': 'relforge1004.eqiad.wmnet', 'port': 9243, 'use_ssl': True}])>, '_remote': <spicerack.remote.Remote object at 0x7fa7e9200100>, '_dry_run': True, '_freeze_writes_index': 'mw_cirrus_metastore', '_freeze_writes_doc_type': 'mw_cirrus_metastore'}
RKemper moved this task from Waiting to Needs Reporting on the Discovery-Search (Current work) board.

Tested with a rolling restart. This ticket is done.