MirrorMaker is now deployed in production, but the jmxtrans metrics we collect are either wrong or incomplete. They have changed in the newer version of Kafka.
Description
Details
Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
Update jmxtrans for confluent kafka mirror maker | operations/puppet | production | +160 -109 |
Event Timeline
Note: We'll have to deal with underscore separated typeNames in the metrics for now. That is, for jmxtrans queries with multiple wildcards, those wildcard matches are concatenated with '_', which makes graphite treat them like a single key in the hierarchy. E.g. MessagesPerSec_kafka-mirror-main-analytics_to_analytics-0_test_otto2, instead of MessagesPerSec.kafka-mirror-main-analytics_to_analytics-0.test_otto2 (metricName, clientId, topic).
The latest version of jmxtrans has a query setting allowDottedKeys that would allow these to be concated with '.' intead of '_', but upgrading would be quite an undertaking at this time.
There is also https://github.com/prometheus/jmx_exporter, which in the future may be the preferred way to monitor JMX beans. Since we already have jmxtrans in place, I'll continue to use it for now.
Change 320424 had a related patch set uploaded (by Ottomata):
Update jmxtrans for confluent kafka mirror maker