Page MenuHomePhabricator

Discerning Cassandra instance in Logstash/Kibana
Closed, ResolvedPublic

Description

For Cassandra log messages displayed in Kibana, there is no indication which instance the message came from; All messages are grouped under the same host. One solution to this would be to add another custom attribute to log messages, parameterized with the instance ID, and on applicable Kibana dashboards (currently only https://logstash.wikimedia.org/#/dashboard/elasticsearch/cassandra), add the field to the "all events" panel.

Event Timeline

GWicke edited projects, added Services (later); removed Services.
Pchelolo raised the priority of this task from Low to Medium.Nov 15 2017, 9:21 AM
Pchelolo subscribed.

Raising the priority since today during the investigation of T180568 this issue have been raised again. In case only one instance per node is misbehaving it's possible to use log stash, but if there would be 2 misbehaving per node that would make log stash absolutely useless.

Change 409916 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] cassandra: add instance ID to list of custom logstash fields

https://gerrit.wikimedia.org/r/409916

Change 409916 merged by Filippo Giunchedi:
[operations/puppet@production] cassandra: add instance ID to list of custom logstash fields

https://gerrit.wikimedia.org/r/409916