Page MenuHomePhabricator

Implement a prometheus exporter for rdkafka in golang
Closed, ResolvedPublic

Description

We currently have two important projects using rdkafka written in golang: purged and atskafka. It would be nice to have the option of exposing rdkafka metrics from those go projects without using an external exporter to get the best of both worlds out of the tradeoffs described in T237993#6104770.

To this end, we should create a go library to parse and expose the metrics returned by rdkafka. Programs written in other languages can take advantage of this library too by using an external exporter that we'll call prometheus-rdkafka-exporter.

Event Timeline

ema created this task.May 20 2020, 8:32 AM
Restricted Application added a project: Operations. · View Herald TranscriptMay 20 2020, 8:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 597574 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Pass JSON file and network address as CLI flags

https://gerrit.wikimedia.org/r/597574

RLazarus triaged this task as Medium priority.May 21 2020, 6:09 PM

Change 597574 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Pass JSON file and network address as CLI flags

https://gerrit.wikimedia.org/r/597574

Change 598006 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Wrap metrics in a struct

https://gerrit.wikimedia.org/r/598006

Change 598007 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Rename update.go to metrics.go

https://gerrit.wikimedia.org/r/598007

Change 598008 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Basic testing with testify

https://gerrit.wikimedia.org/r/598008

Change 598009 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Packaging and copyring notices

https://gerrit.wikimedia.org/r/598009

Change 598016 had a related patch set uploaded (by Ema; owner: Ema):
[integration/config@master] Test prometheus-rdkafka-exporter with debian-glue-backports

https://gerrit.wikimedia.org/r/598016

Change 598016 merged by jenkins-bot:
[integration/config@master] Test prometheus-rdkafka-exporter with debian-glue-backports

https://gerrit.wikimedia.org/r/598016

Change 598006 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Wrap metrics in a struct

https://gerrit.wikimedia.org/r/598006

Change 598007 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Rename update.go to metrics.go

https://gerrit.wikimedia.org/r/598007

Change 598008 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Basic testing with testify

https://gerrit.wikimedia.org/r/598008

Change 598009 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Packaging and copyright notices

https://gerrit.wikimedia.org/r/598009

Change 598420 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Move consumer/producer JSON files to fixture directory

https://gerrit.wikimedia.org/r/598420

Change 598421 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Test Update with actual producer/consumer stats

https://gerrit.wikimedia.org/r/598421

Change 598422 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Fix panic on prometheus.GaugeVec label cardinality mismatch

https://gerrit.wikimedia.org/r/598422

Change 598420 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Move consumer/producer JSON files to fixture directory

https://gerrit.wikimedia.org/r/598420

Change 598421 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Test Update with actual producer/consumer stats

https://gerrit.wikimedia.org/r/598421

Change 598422 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Fix panic on prometheus.GaugeVec label cardinality mismatch

https://gerrit.wikimedia.org/r/598422

Mentioned in SAL (#wikimedia-operations) [2020-05-25T11:01:35Z] <ema> upload prometheus-rdkafka-exporter to buster-wikimedia T253197

Change 598444 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/atskafka@master] 0.7: use prometheus-rdkafka-exporter

https://gerrit.wikimedia.org/r/598444

ema closed this task as Resolved.May 25 2020, 12:17 PM
ema claimed this task.

The package prometheus-rdkafka-exporter is now available in buster-wikimedia, closing.

Change 598444 merged by Ema:
[operations/software/atskafka@master] 0.7: use prometheus-rdkafka-exporter

https://gerrit.wikimedia.org/r/598444