Page MenuHomePhabricator

Update Kafka MirrorMaker jmxtrans attributes and make MirrorMaker grafana dashboard
Closed, ResolvedPublic5 Estimated Story Points

Description

MirrorMaker is now deployed in production, but the jmxtrans metrics we collect are either wrong or incomplete. They have changed in the newer version of Kafka.

Event Timeline

Milimetric triaged this task as Medium priority.Aug 22 2016, 3:34 PM
Milimetric moved this task from Incoming to Dashiki on the Analytics board.
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.
Ottomata set the point value for this task to 8.

Note: We'll have to deal with underscore separated typeNames in the metrics for now. That is, for jmxtrans queries with multiple wildcards, those wildcard matches are concatenated with '_', which makes graphite treat them like a single key in the hierarchy. E.g. MessagesPerSec_kafka-mirror-main-analytics_to_analytics-0_test_otto2, instead of MessagesPerSec.kafka-mirror-main-analytics_to_analytics-0.test_otto2 (metricName, clientId, topic).

The latest version of jmxtrans has a query setting allowDottedKeys that would allow these to be concated with '.' intead of '_', but upgrading would be quite an undertaking at this time.

There is also https://github.com/prometheus/jmx_exporter, which in the future may be the preferred way to monitor JMX beans. Since we already have jmxtrans in place, I'll continue to use it for now.

Change 320424 had a related patch set uploaded (by Ottomata):
Update jmxtrans for confluent kafka mirror maker

https://gerrit.wikimedia.org/r/320424

Change 320424 merged by Ottomata:
Update jmxtrans for confluent kafka mirror maker

https://gerrit.wikimedia.org/r/320424

Ottomata changed the point value for this task from 8 to 5.Nov 8 2016, 10:12 PM
Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.