Page MenuHomePhabricator
Create Task
Maniphest T198351

Refactor puppet to support multiple elasticsearch instances on same node
Closed, ResolvedPublic
Actions

Details

SubjectRepoBranchLines +/-
elasticsearch: default instance config for deployment-prepoperations/puppetproduction+1 -0
Allow multiple elasticsearch instances per hostoperations/puppetproduction+60 -40
tools: Update usage of ::elasticsearch (take 2)operations/puppetproduction+37 -37
tools: Update usage of ::elasticsearchoperations/puppetproduction+6 -1
Split instance define out of elasticsearch classoperations/puppetproduction+828 -694
prometheus/elasticsearch support multiple exporters per hostoperations/puppetproduction+143 -49
convert role::logstash::elasticsearch to profilesoperations/puppetproduction+137 -92
elasticsearch: load jvm options from instance specific config diroperations/puppetproduction+1 -0
elasticsearch: disable the default elasticsearch unitoperations/puppetproduction+13 -73
Convert elasticsearch to systemd unitoperations/puppetproduction+115 -42
elasticsearch: move elasticsearch data directoryoperations/puppetproduction+1 -0
elasticsearch: move elasticsearch data directoryoperations/puppetproduction+1 -0
logstash: move elasticsearch data directoryoperations/puppetproduction+1 -1
logstash: move elasticsearch data directoryoperations/puppetproduction+1 -1
relforge: move elasticsearch data directoryoperations/puppetproduction+1 -0
Switch elasticsearch to use tlsproxy moduleoperations/puppetproduction+56 -123
Make elasticsearch http and transport ports explicitoperations/puppetproduction+43 -19
elasticsearch: new SSL cert for search.svc.eqiad.wmnetoperations/puppetproduction+26 -0
elasticsearch: new SSL cert for search.svc.codfw.wmnetoperations/puppetproduction+26 -0
elasticsearch: new SSL cert for relforgeoperations/puppetproduction+26 -0
Make cirrus specific elasticsearch profileoperations/puppetproduction+72 -62
Split elasticsearch::log::hot_threads into two piecesoperations/puppetproduction+77 -22
Cleanup ensure => absent after refactoringoperations/puppetproduction+6 -4
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

EBernhardson triaged this task as Medium priority.Jun 27 2018, 7:31 PM
EBernhardson created this task.
EBernhardson moved this task from needs triage to Current work on the Discovery-Search board.
EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.
EBernhardson moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.
EBernhardson added a subtask: T198490: Use kafka for communication from analytics cluster to elasticsearch.Jul 10 2018, 5:22 PM
EBernhardson moved this task from Needs review to not in use - please delete on the Discovery-Search (Current work) board.Jul 23 2018, 5:18 PM
gerritbot added a subscriber: gerritbot.Jul 23 2018, 11:49 PM

Change 444610 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Switch elasticsearch to use tlsproxy module

https://gerrit.wikimedia.org/r/444610

gerritbot added a project: Patch-For-Review.Jul 23 2018, 11:49 PM

Change 440498 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Prep work for multi-instance elasticsearch refactor

https://gerrit.wikimedia.org/r/440498

gerritbot added a comment.Jul 23 2018, 11:49 PM

Change 441894 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] convert role::logstash::elasticsearch to profiles

https://gerrit.wikimedia.org/r/441894

gerritbot added a comment.Jul 23 2018, 11:49 PM

Change 447565 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Split elasticsearch::log::hot_threads into two pieces

https://gerrit.wikimedia.org/r/447565

gerritbot added a comment.Jul 23 2018, 11:49 PM

Change 447566 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Make cirrus specific elasticsearch profile

https://gerrit.wikimedia.org/r/447566

gerritbot added a comment.Jul 23 2018, 11:49 PM

Change 447568 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Make elasticsearch http and transport ports explicit

https://gerrit.wikimedia.org/r/447568

gerritbot added a comment.Jul 23 2018, 11:53 PM

Change 441321 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] prometheus/elasticsearch support multiple exporters per host

https://gerrit.wikimedia.org/r/441321

gerritbot added a comment.Jul 23 2018, 11:53 PM

Change 441338 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Split instance define out of elasticsearch class

https://gerrit.wikimedia.org/r/441338

gerritbot added a comment.Jul 23 2018, 11:53 PM

Change 440049 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Allow multiple elasticsearch instances per host

https://gerrit.wikimedia.org/r/440049

gerritbot added a comment.Jul 23 2018, 11:53 PM

Change 444765 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Cleanup ensure => absent after refactoring

https://gerrit.wikimedia.org/r/444765

gerritbot added a comment.Jul 25 2018, 11:21 PM

Change 444765 abandoned by EBernhardson:
Cleanup ensure => absent after refactoring

Reason:
going to use a cleanup script and remove resources as we go instead of afterwards

https://gerrit.wikimedia.org/r/444765

gerritbot added a comment.Jul 30 2018, 9:47 AM

Change 447565 merged by Gehel:
[operations/puppet@production] Split elasticsearch::log::hot_threads into two pieces

https://gerrit.wikimedia.org/r/447565

gerritbot added a comment.Jul 30 2018, 11:31 AM

Change 447566 merged by Gehel:
[operations/puppet@production] Make cirrus specific elasticsearch profile

https://gerrit.wikimedia.org/r/447566

EBernhardson moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Aug 21 2018, 5:36 PM
Gehel claimed this task.Aug 21 2018, 5:37 PM

I'm taking over this task to review and deploy the remaining patches.

Stashbot added a subscriber: Stashbot.Aug 23 2018, 9:17 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-23T09:17:11Z] <gehel> starting to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/444610 as part of T198351, including regeneration of SSL certs. Disabling puppet on elastic* during the operation

gerritbot added a comment.Aug 23 2018, 9:40 AM

Change 444610 merged by Gehel:
[operations/puppet@production] Switch elasticsearch to use tlsproxy module

https://gerrit.wikimedia.org/r/444610

gerritbot added a comment.Aug 23 2018, 9:45 AM

Change 454778 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: new SSL cert for relforge

https://gerrit.wikimedia.org/r/454778

gerritbot added a comment.Aug 23 2018, 9:46 AM

Change 454778 merged by Gehel:
[operations/puppet@production] elasticsearch: new SSL cert for relforge

https://gerrit.wikimedia.org/r/454778

gerritbot added a comment.Aug 23 2018, 10:05 AM

Change 454780 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: new SSL cert for search.svc.codfw.wmnet

https://gerrit.wikimedia.org/r/454780

gerritbot added a comment.Aug 23 2018, 10:06 AM

Change 454780 merged by Gehel:
[operations/puppet@production] elasticsearch: new SSL cert for search.svc.codfw.wmnet

https://gerrit.wikimedia.org/r/454780

gerritbot added a comment.Aug 23 2018, 10:30 AM

Change 454788 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: new SSL cert for search.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/454788

gerritbot added a comment.Aug 23 2018, 10:31 AM

Change 454788 merged by Gehel:
[operations/puppet@production] elasticsearch: new SSL cert for search.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/454788

Stashbot added a comment.Aug 23 2018, 11:00 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-23T11:00:29Z] <gehel> new SSL certs / tlsproxy deployed on elastic nodes - T198351

gerritbot added a comment.Aug 23 2018, 2:55 PM

Change 447568 merged by Gehel:
[operations/puppet@production] Make elasticsearch http and transport ports explicit

https://gerrit.wikimedia.org/r/447568

gerritbot added a comment.Aug 29 2018, 1:08 PM

Change 456133 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456133

gerritbot added a comment.Aug 29 2018, 1:10 PM

Change 456135 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] relforge: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456135

gerritbot added a comment.Aug 29 2018, 1:13 PM

Change 456137 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456137

gerritbot added a comment.Aug 29 2018, 1:13 PM

Change 456138 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456138

gerritbot added a comment.Aug 29 2018, 1:35 PM

Change 456135 merged by Gehel:
[operations/puppet@production] relforge: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456135

Stashbot added a comment.Aug 30 2018, 4:15 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-30T16:15:05Z] <gehel> restart of logstash to move data directory - T198351

gerritbot added a comment.Aug 30 2018, 4:17 PM

Change 456133 merged by Gehel:
[operations/puppet@production] logstash: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456133

Stashbot added a comment.Aug 30 2018, 4:29 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-30T16:29:44Z] <gehel> shutting down wdqs1005 for new SSD and reimaging - T198351

gerritbot added a comment.Aug 31 2018, 8:44 AM

Change 456576 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456576

gerritbot added a comment.Aug 31 2018, 8:52 AM

Change 456576 merged by Gehel:
[operations/puppet@production] logstash: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456576

Stashbot added a comment.Sep 3 2018, 9:00 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-03T09:00:25Z] <gehel> starting rolling restart of elasticsearch / cirrus / codfw for various updates and data directory migration - T198351

gerritbot added a comment.Sep 3 2018, 9:04 AM

Change 456137 merged by Gehel:
[operations/puppet@production] elasticsearch: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456137

Stashbot added a comment.Sep 4 2018, 7:52 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-04T07:52:00Z] <gehel> rolling restart of elasticsearch / cirrus / codfw for various updates and data directory migration completed - T198351

Stashbot added a comment.Sep 4 2018, 8:28 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-04T08:28:40Z] <gehel> starting rolling restart of elasticsearch / cirrus / eqiad for various updates and data directory migration - T198351

gerritbot added a comment.Sep 4 2018, 8:30 AM

Change 456138 merged by Gehel:
[operations/puppet@production] elasticsearch: move elasticsearch data directory

https://gerrit.wikimedia.org/r/456138

Stashbot added a comment.Sep 4 2018, 7:26 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-04T19:26:00Z] <gehel> rolling restart of elasticsearch / cirrus / eqiad for various updates and data directory migration completed - T198351

EBernhardson added a comment.Sep 4 2018, 8:31 PM

@Gehel looks like the last one that needs it's data directory migrated is logstash. Although logstash doesn't need the distinction, the shared systemd unit in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/440498/ will require it.

EBernhardson added a comment.Sep 4 2018, 8:33 PM

Strike that, i simply failed to look at the latest version of puppet. It all looks great and ready for the systemd migration next

gerritbot added a comment.Sep 6 2018, 8:59 AM

Change 458464 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: disable the default elasticsearch unit

https://gerrit.wikimedia.org/r/458464

debt closed subtask T198490: Use kafka for communication from analytics cluster to elasticsearch as Resolved.Sep 13 2018, 9:22 PM
gerritbot added a comment.Sep 19 2018, 12:08 PM

Change 440498 merged by Gehel:
[operations/puppet@production] Convert elasticsearch to systemd unit

https://gerrit.wikimedia.org/r/440498

gerritbot added a comment.Sep 21 2018, 8:46 AM

Change 458464 merged by Gehel:
[operations/puppet@production] elasticsearch: disable the default elasticsearch unit

https://gerrit.wikimedia.org/r/458464

gerritbot added a comment.Sep 21 2018, 9:36 AM

Change 461926 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: load jvm options from instance specific config dir

https://gerrit.wikimedia.org/r/461926

gerritbot added a comment.Sep 21 2018, 10:04 AM

Change 461926 merged by Gehel:
[operations/puppet@production] elasticsearch: load jvm options from instance specific config dir

https://gerrit.wikimedia.org/r/461926

fgiunchedi mentioned this in T203169: Logstash hardware expansion.Sep 24 2018, 1:49 PM
gerritbot added a comment.Sep 25 2018, 7:34 AM

Change 441894 merged by Gehel:
[operations/puppet@production] convert role::logstash::elasticsearch to profiles

https://gerrit.wikimedia.org/r/441894

gerritbot added a comment.Sep 25 2018, 4:33 PM

Change 441321 merged by Gehel:
[operations/puppet@production] prometheus/elasticsearch support multiple exporters per host

https://gerrit.wikimedia.org/r/441321

gerritbot added a comment.Sep 26 2018, 12:28 PM

Change 441338 merged by Gehel:
[operations/puppet@production] Split instance define out of elasticsearch class

https://gerrit.wikimedia.org/r/441338

gerritbot added a comment.Sep 27 2018, 9:58 PM

Change 463385 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] tools: Update usage of ::elasticsearch

https://gerrit.wikimedia.org/r/463385

gerritbot added a comment.Sep 27 2018, 10:02 PM

Change 463385 merged by Andrew Bogott:
[operations/puppet@production] tools: Update usage of ::elasticsearch

https://gerrit.wikimedia.org/r/463385

gerritbot added a comment.Sep 27 2018, 10:21 PM

Change 463386 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] tools: Update usage of ::elasticsearch (take 2)

https://gerrit.wikimedia.org/r/463386

gerritbot added a comment.Sep 28 2018, 3:08 AM

Change 463386 merged by Andrew Bogott:
[operations/puppet@production] tools: Update usage of ::elasticsearch (take 2)

https://gerrit.wikimedia.org/r/463386

gerritbot added a comment.Oct 1 2018, 1:51 PM

Change 440049 merged by Gehel:
[operations/puppet@production] Allow multiple elasticsearch instances per host

https://gerrit.wikimedia.org/r/440049

gerritbot added a comment.Oct 1 2018, 1:57 PM

Change 463761 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: default instance config for deployment-prep

https://gerrit.wikimedia.org/r/463761

gerritbot added a comment.Oct 1 2018, 1:59 PM

Change 463761 merged by Gehel:
[operations/puppet@production] elasticsearch: default instance config for deployment-prep

https://gerrit.wikimedia.org/r/463761

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Oct 1 2018, 8:50 PM
debt closed this task as Resolved.Oct 5 2018, 3:57 PM
colewhite mentioned this in T301382: Set up opensearch cluster for datahub.Mar 3 2022, 4:04 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL