Page MenuHomePhabricator
Create Task
Maniphest T278378

Pull Elasticsearch config out of Spicerack
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Any changes to spicerack require a manual deploy, which is painful. Move the Elasticsearch cluster config into puppet.

This is where the config that needs to be pulled out lives: https://gerrit.wikimedia.org/g/operations/software/spicerack/+/master/spicerack/elasticsearch_cluster.py#23

Todo:

  • Create a YAML spicerack configuration file that mimics the current config constant. Spicerack configurations are managed in puppet. This configuration can be static at first, but ideally it should be created from state already in puppet (I'm not sure if this is easily doable - @EBernhardson might have some ideas)
  • In Spicerack, replace the configuration constant with the same data, but loaded from a configuration file that is passed to the elasticsearch_cluster module. See other examples of load_yaml_config in the Spicerack class.

Details

SubjectRepoBranchLines +/-
elasticsearch: move cluster configuration to puppetoperations/puppetproduction+32 -0
elasticsearch: load config from yamloperations/software/spicerackmaster+14 -28
Customize query in gerrit

Related Objects
Search...

Event Timeline

RKemper created this task.Mar 24 2021, 7:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 24 2021, 7:28 PM
RKemper updated the task description. (Show Details)Mar 24 2021, 7:30 PM
Gehel set the point value for this task to 3.Mar 29 2021, 3:30 PM
Gehel moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
Gehel updated the task description. (Show Details)Oct 11 2021, 7:24 AM
Gehel updated the task description. (Show Details)
Gehel added a subscriber: EBernhardson.
Gehel updated the task description. (Show Details)Oct 11 2021, 7:26 AM
Gehel updated the task description. (Show Details)
gerritbot added a comment.Oct 13 2021, 4:25 PM

Change 716532 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/software/spicerack@master] elasticsearch: begin pull out spicerack config

https://gerrit.wikimedia.org/r/716532

gerritbot added a project: Patch-For-Review.Oct 13 2021, 4:25 PM
RKemper added a comment.Oct 15 2021, 7:16 PM

Some context on the general plan from my convo with SDineshKumar (who is going to take a swing at this)

(1) Change spicerack so that it is still hardcoded, but instead of directly hardcoding the config in the python dictionary as it does currently, have it hardcode the contents of the config in a yaml file and parse that file
(At this point spicerack will work the same but now we've changed the internals so that it's reading a YAML file, except instead of reading an actual file we're "mocking" the file read by having the contents of that file hardcoded in a variable in the code)
(2) Deploy puppet change to actually write the file to disk on `cumin*` hosts (cumin hosts are where we run cookbooks from)
(At this point the puppet code is writing the file but spicerack isn't reading it, so it's effectively a no-op but we can verify that the file is where it should be)
(3) Change spicerack to read from the actual file on disk rather than mock it. This should be minimal code change because we've already refactored in step #1
So step #3 is the only point at which we could break something, but it should just work if we did steps #1 and #2 properly
TheReadOnly subscribed.Oct 26 2021, 6:25 PM

Hi @RKemper

Due to family situations, I couldn't pick this up yet. I will pick this up by 31st Oct.

Thanks
Dinesh

Gehel moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.Feb 16 2022, 8:49 PM
Gehel moved this task from In Progress to Waiting on the Discovery-Search (Current work) board.Feb 22 2022, 8:18 PM
Gehel added a parent task: T301955: Upgrade relforge to elasticsearch 6.8.23.Mar 7 2022, 1:02 PM
Gehel mentioned this in T301955: Upgrade relforge to elasticsearch 6.8.23.Mar 7 2022, 1:04 PM
gerritbot added a comment.Mar 7 2022, 10:19 PM

Change 768816 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] elasticsearch: move cluster configuration to puppet

https://gerrit.wikimedia.org/r/768816

gerritbot added a comment.Mar 8 2022, 5:58 PM

Change 768816 merged by Bking:

[operations/puppet@production] elasticsearch: move cluster configuration to puppet

https://gerrit.wikimedia.org/r/768816

gerritbot added a comment.Mar 9 2022, 10:40 AM

Change 716532 merged by Volans:

[operations/software/spicerack@master] elasticsearch: load config from yaml

https://gerrit.wikimedia.org/r/716532

Maintenance_bot removed a project: Patch-For-Review.Mar 9 2022, 11:10 AM
• razzi closed this task as Resolved.Mar 9 2022, 5:00 PM
• razzi subscribed.

This is done! Note that the relforge host has been updated to relforge1004

razzi@cumin1001:~$ sudo python3
>>> from spicerack import Spicerack
>>> s = Spicerack(verbose=True, dry_run=True)
>>> ec.__dict__
KeyboardInterrupt
>>> ec=s.elasticsearch_clusters("relforge", ("some_core_dc",))
>>> ec.__dict__['_clusters'][0].__dict__
{'_elasticsearch': <Elasticsearch([{'host': 'relforge1004.eqiad.wmnet', 'port': 9243, 'use_ssl': True}])>, '_remote': <spicerack.remote.Remote object at 0x7fa7e9200100>, '_dry_run': True, '_freeze_writes_index': 'mw_cirrus_metastore', '_freeze_writes_doc_type': 'mw_cirrus_metastore'}
RKemper reopened this task as Open.Mar 9 2022, 9:27 PM
RKemper moved this task from Waiting to Needs Reporting on the Discovery-Search (Current work) board.

Tested with a rolling restart. This ticket is done.

Gehel closed this task as Resolved.Mar 14 2022, 2:52 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits