Page MenuHomePhabricator

Refactor puppet to support multiple elasticsearch instances on same node
Closed, ResolvedPublic

Details

Related Gerrit Patches:
operations/puppet : productionelasticsearch: default instance config for deployment-prep
operations/puppet : productionAllow multiple elasticsearch instances per host
operations/puppet : productiontools: Update usage of ::elasticsearch (take 2)
operations/puppet : productiontools: Update usage of ::elasticsearch
operations/puppet : productionSplit instance define out of elasticsearch class
operations/puppet : productionprometheus/elasticsearch support multiple exporters per host
operations/puppet : productionconvert role::logstash::elasticsearch to profiles
operations/puppet : productionelasticsearch: load jvm options from instance specific config dir
operations/puppet : productionelasticsearch: disable the default elasticsearch unit
operations/puppet : productionConvert elasticsearch to systemd unit
operations/puppet : productionelasticsearch: move elasticsearch data directory
operations/puppet : productionelasticsearch: move elasticsearch data directory
operations/puppet : productionlogstash: move elasticsearch data directory
operations/puppet : productionlogstash: move elasticsearch data directory
operations/puppet : productionrelforge: move elasticsearch data directory
operations/puppet : productionSwitch elasticsearch to use tlsproxy module
operations/puppet : productionMake elasticsearch http and transport ports explicit
operations/puppet : productionelasticsearch: new SSL cert for search.svc.eqiad.wmnet
operations/puppet : productionelasticsearch: new SSL cert for search.svc.codfw.wmnet
operations/puppet : productionelasticsearch: new SSL cert for relforge
operations/puppet : productionMake cirrus specific elasticsearch profile
operations/puppet : productionSplit elasticsearch::log::hot_threads into two pieces
operations/puppet : productionCleanup ensure => absent after refactoring

Event Timeline

EBernhardson triaged this task as Medium priority.Jun 27 2018, 7:31 PM
EBernhardson created this task.
EBernhardson moved this task from needs triage to Current work on the Discovery-Search board.

Change 444610 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Switch elasticsearch to use tlsproxy module

https://gerrit.wikimedia.org/r/444610

Change 440498 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Prep work for multi-instance elasticsearch refactor

https://gerrit.wikimedia.org/r/440498

Change 441894 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] convert role::logstash::elasticsearch to profiles

https://gerrit.wikimedia.org/r/441894

Change 447565 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Split elasticsearch::log::hot_threads into two pieces

https://gerrit.wikimedia.org/r/447565

Change 447566 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Make cirrus specific elasticsearch profile

https://gerrit.wikimedia.org/r/447566

Change 447568 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Make elasticsearch http and transport ports explicit

https://gerrit.wikimedia.org/r/447568

Change 441321 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] prometheus/elasticsearch support multiple exporters per host

https://gerrit.wikimedia.org/r/441321

Change 441338 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Split instance define out of elasticsearch class

https://gerrit.wikimedia.org/r/441338

Change 440049 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Allow multiple elasticsearch instances per host

https://gerrit.wikimedia.org/r/440049

Change 444765 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Cleanup ensure => absent after refactoring

https://gerrit.wikimedia.org/r/444765

Change 444765 abandoned by EBernhardson:
Cleanup ensure => absent after refactoring

Reason:
going to use a cleanup script and remove resources as we go instead of afterwards

https://gerrit.wikimedia.org/r/444765

Change 447565 merged by Gehel:
[operations/puppet@production] Split elasticsearch::log::hot_threads into two pieces

https://gerrit.wikimedia.org/r/447565

Change 447566 merged by Gehel:
[operations/puppet@production] Make cirrus specific elasticsearch profile

https://gerrit.wikimedia.org/r/447566

Gehel claimed this task.Aug 21 2018, 5:37 PM

I'm taking over this task to review and deploy the remaining patches.

Mentioned in SAL (#wikimedia-operations) [2018-08-23T09:17:11Z] <gehel> starting to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/444610 as part of T198351, including regeneration of SSL certs. Disabling puppet on elastic* during the operation

Change 444610 merged by Gehel:
[operations/puppet@production] Switch elasticsearch to use tlsproxy module

https://gerrit.wikimedia.org/r/444610

Change 454778 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: new SSL cert for relforge

https://gerrit.wikimedia.org/r/454778

Change 454778 merged by Gehel:
[operations/puppet@production] elasticsearch: new SSL cert for relforge

https://gerrit.wikimedia.org/r/454778

Change 454780 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: new SSL cert for search.svc.codfw.wmnet

https://gerrit.wikimedia.org/r/454780

Change 454780 merged by Gehel:
[operations/puppet@production] elasticsearch: new SSL cert for search.svc.codfw.wmnet

https://gerrit.wikimedia.org/r/454780

Change 454788 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: new SSL cert for search.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/454788

Change 454788 merged by Gehel:
[operations/puppet@production] elasticsearch: new SSL cert for search.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/454788

Mentioned in SAL (#wikimedia-operations) [2018-08-23T11:00:29Z] <gehel> new SSL certs / tlsproxy deployed on elastic nodes - T198351

Change 447568 merged by Gehel:
[operations/puppet@production] Make elasticsearch http and transport ports explicit

https://gerrit.wikimedia.org/r/447568

Change 456133 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456133

Change 456135 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] relforge: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456135

Change 456137 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456137

Change 456138 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456138

Change 456135 merged by Gehel:
[operations/puppet@production] relforge: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456135

Mentioned in SAL (#wikimedia-operations) [2018-08-30T16:15:05Z] <gehel> restart of logstash to move data directory - T198351

Change 456133 merged by Gehel:
[operations/puppet@production] logstash: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456133

Mentioned in SAL (#wikimedia-operations) [2018-08-30T16:29:44Z] <gehel> shutting down wdqs1005 for new SSD and reimaging - T198351

Change 456576 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456576

Change 456576 merged by Gehel:
[operations/puppet@production] logstash: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456576

Mentioned in SAL (#wikimedia-operations) [2018-09-03T09:00:25Z] <gehel> starting rolling restart of elasticsearch / cirrus / codfw for various updates and data directory migration - T198351

Change 456137 merged by Gehel:
[operations/puppet@production] elasticsearch: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456137

Mentioned in SAL (#wikimedia-operations) [2018-09-04T07:52:00Z] <gehel> rolling restart of elasticsearch / cirrus / codfw for various updates and data directory migration completed - T198351

Mentioned in SAL (#wikimedia-operations) [2018-09-04T08:28:40Z] <gehel> starting rolling restart of elasticsearch / cirrus / eqiad for various updates and data directory migration - T198351

Change 456138 merged by Gehel:
[operations/puppet@production] elasticsearch: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456138

Mentioned in SAL (#wikimedia-operations) [2018-09-04T19:26:00Z] <gehel> rolling restart of elasticsearch / cirrus / eqiad for various updates and data directory migration completed - T198351

@Gehel looks like the last one that needs it's data directory migrated is logstash. Although logstash doesn't need the distinction, the shared systemd unit in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/440498/ will require it.

Strike that, i simply failed to look at the latest version of puppet. It all looks great and ready for the systemd migration next

Change 458464 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: disable the default elasticsearch unit

https://gerrit.wikimedia.org/r/458464

Change 440498 merged by Gehel:
[operations/puppet@production] Convert elasticsearch to systemd unit

https://gerrit.wikimedia.org/r/440498

Change 458464 merged by Gehel:
[operations/puppet@production] elasticsearch: disable the default elasticsearch unit

https://gerrit.wikimedia.org/r/458464

Change 461926 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: load jvm options from instance specific config dir

https://gerrit.wikimedia.org/r/461926

Change 461926 merged by Gehel:
[operations/puppet@production] elasticsearch: load jvm options from instance specific config dir

https://gerrit.wikimedia.org/r/461926

Change 441894 merged by Gehel:
[operations/puppet@production] convert role::logstash::elasticsearch to profiles

https://gerrit.wikimedia.org/r/441894

Change 441321 merged by Gehel:
[operations/puppet@production] prometheus/elasticsearch support multiple exporters per host

https://gerrit.wikimedia.org/r/441321

Change 441338 merged by Gehel:
[operations/puppet@production] Split instance define out of elasticsearch class

https://gerrit.wikimedia.org/r/441338

Change 463385 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] tools: Update usage of ::elasticsearch

https://gerrit.wikimedia.org/r/463385

Change 463385 merged by Andrew Bogott:
[operations/puppet@production] tools: Update usage of ::elasticsearch

https://gerrit.wikimedia.org/r/463385

Change 463386 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] tools: Update usage of ::elasticsearch (take 2)

https://gerrit.wikimedia.org/r/463386

Change 463386 merged by Andrew Bogott:
[operations/puppet@production] tools: Update usage of ::elasticsearch (take 2)

https://gerrit.wikimedia.org/r/463386

Change 440049 merged by Gehel:
[operations/puppet@production] Allow multiple elasticsearch instances per host

https://gerrit.wikimedia.org/r/440049

Change 463761 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: default instance config for deployment-prep

https://gerrit.wikimedia.org/r/463761

Change 463761 merged by Gehel:
[operations/puppet@production] elasticsearch: default instance config for deployment-prep

https://gerrit.wikimedia.org/r/463761

debt closed this task as Resolved.Oct 5 2018, 3:57 PM