Page MenuHomePhabricator

Update Logstash Puppet roles to allow separating Elasticsearch from Logstash & Kibana
Closed, ResolvedPublic

Description

Once T96692: Rack and Setup (3) Logstash Servers is complete we can start setting up the new servers as Elasticsearch backends for the existing logstash100[1-3] boxes. The Puppet roles will need to be refactored to have frontend (logstash, kibana) and backend (elasticsearch) roles that can be applied separately to the two classes of hosts in the new larger cluster.

Communication between Logstash and the Elasticsearch cluster will be easier to manage if the frontend role hosts include a client only Elasticsearch node (node.master: false; node.data: false) that joins the backend server cluster. This will allow Logstash to continue to communicate with http://localhost:9200/ and then use the robust cluster communication routing provided by Elasticsearch to find the right data node to store the log events.

Event Timeline

bd808 claimed this task.
bd808 raised the priority of this task from to High.
bd808 updated the task description. (Show Details)
bd808 added a project: Wikimedia-Logstash.
bd808 added a subscriber: bd808.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 22 2015, 3:48 AM

Change 205970 had a related patch set uploaded (by BryanDavis):
logstash: Provision Elasticsearch only backend hosts

https://gerrit.wikimedia.org/r/205970

Change 205971 had a related patch set uploaded (by BryanDavis):
logstash: Convert Elasticsearch on logstash100[1-3] to client

https://gerrit.wikimedia.org/r/205971

Change 205970 merged by Gage:
logstash: Provision Elasticsearch only backend hosts

https://gerrit.wikimedia.org/r/205970

bd808 moved this task from To Do to Needs Review/Feedback on the User-bd808 board.

Change 205971 merged by Yuvipanda:
logstash: Convert Elasticsearch on logstash100[1-3] to client

https://gerrit.wikimedia.org/r/205971

bd808 closed this task as Resolved.May 6 2015, 2:14 AM
bd808 moved this task from Needs Review/Feedback to Done on the User-bd808 board.
bd808 moved this task from Done to Archive on the User-bd808 board.May 12 2015, 6:23 AM
bd808 moved this task from Backlog to Archive on the Wikimedia-Logstash board.Jun 5 2015, 6:55 PM