Page MenuHomePhabricator
Create Task
Maniphest T253197

Implement a prometheus exporter for rdkafka in golang
Closed, ResolvedPublic
Actions

Description

We currently have two important projects using rdkafka written in golang: purged and atskafka. It would be nice to have the option of exposing rdkafka metrics from those go projects without using an external exporter to get the best of both worlds out of the tradeoffs described in T237993#6104770.

To this end, we should create a go library to parse and expose the metrics returned by rdkafka. Programs written in other languages can take advantage of this library too by using an external exporter that we'll call prometheus-rdkafka-exporter.

Details

SubjectRepoBranchLines +/-
0.7: use prometheus-rdkafka-exporteroperations/software/atskafkamaster+25 -20
Fix panic on prometheus.GaugeVec label cardinality mismatchoperations/software/prometheus-rdkafka-exportermaster+15 -2
Test Update with actual producer/consumer statsoperations/software/prometheus-rdkafka-exportermaster+21 -2
Move consumer/producer JSON files to fixture directoryoperations/software/prometheus-rdkafka-exportermaster+0 -0
Packaging and copyright noticesoperations/software/prometheus-rdkafka-exportermaster+98 -0
Basic testing with testifyoperations/software/prometheus-rdkafka-exportermaster+39 -0
Rename update.go to metrics.gooperations/software/prometheus-rdkafka-exportermaster+0 -0
Wrap metrics in a structoperations/software/prometheus-rdkafka-exportermaster+25 -15
Test prometheus-rdkafka-exporter with debian-glue-backportsintegration/configmaster+1 -0
Pass JSON file and network address as CLI flagsoperations/software/prometheus-rdkafka-exportermaster+8 -3
Customize query in gerrit

Related Objects

Event Timeline

• ema created this task.May 20 2020, 8:32 AM
Restricted Application added a project: SRE. · View Herald TranscriptMay 20 2020, 8:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• ema mentioned this in T237993: Create replacement for Varnishkafka.May 20 2020, 8:35 AM
gerritbot added a comment.May 20 2020, 3:38 PM

Change 597574 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Pass JSON file and network address as CLI flags

https://gerrit.wikimedia.org/r/597574

gerritbot added a project: Patch-For-Review.May 20 2020, 3:38 PM
RLazarus triaged this task as Medium priority.May 21 2020, 6:09 PM
gerritbot added a comment.May 22 2020, 8:34 AM

Change 597574 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Pass JSON file and network address as CLI flags

https://gerrit.wikimedia.org/r/597574

Maintenance_bot removed a project: Patch-For-Review.May 22 2020, 9:10 AM
gerritbot added a comment.May 22 2020, 10:17 AM

Change 598006 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Wrap metrics in a struct

https://gerrit.wikimedia.org/r/598006

gerritbot added a project: Patch-For-Review.May 22 2020, 10:17 AM

Change 598007 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Rename update.go to metrics.go

https://gerrit.wikimedia.org/r/598007

gerritbot added a comment.May 22 2020, 10:17 AM

Change 598008 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Basic testing with testify

https://gerrit.wikimedia.org/r/598008

gerritbot added a comment.May 22 2020, 10:17 AM

Change 598009 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Packaging and copyring notices

https://gerrit.wikimedia.org/r/598009

gerritbot added a comment.May 22 2020, 10:49 AM

Change 598016 had a related patch set uploaded (by Ema; owner: Ema):
[integration/config@master] Test prometheus-rdkafka-exporter with debian-glue-backports

https://gerrit.wikimedia.org/r/598016

gerritbot added a comment.May 22 2020, 1:30 PM

Change 598016 merged by jenkins-bot:
[integration/config@master] Test prometheus-rdkafka-exporter with debian-glue-backports

https://gerrit.wikimedia.org/r/598016

gerritbot added a comment.May 22 2020, 1:58 PM

Change 598006 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Wrap metrics in a struct

https://gerrit.wikimedia.org/r/598006

gerritbot added a comment.May 22 2020, 1:58 PM

Change 598007 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Rename update.go to metrics.go

https://gerrit.wikimedia.org/r/598007

gerritbot added a comment.May 22 2020, 2:33 PM

Change 598008 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Basic testing with testify

https://gerrit.wikimedia.org/r/598008

gerritbot added a comment.May 22 2020, 2:34 PM

Change 598009 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Packaging and copyright notices

https://gerrit.wikimedia.org/r/598009

Maintenance_bot removed a project: Patch-For-Review.May 22 2020, 3:10 PM
gerritbot added a comment.May 25 2020, 8:58 AM

Change 598420 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Move consumer/producer JSON files to fixture directory

https://gerrit.wikimedia.org/r/598420

gerritbot added a project: Patch-For-Review.May 25 2020, 8:58 AM

Change 598421 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Test Update with actual producer/consumer stats

https://gerrit.wikimedia.org/r/598421

gerritbot added a comment.May 25 2020, 8:58 AM

Change 598422 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] Fix panic on prometheus.GaugeVec label cardinality mismatch

https://gerrit.wikimedia.org/r/598422

gerritbot added a comment.May 25 2020, 8:59 AM

Change 598420 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Move consumer/producer JSON files to fixture directory

https://gerrit.wikimedia.org/r/598420

gerritbot added a comment.May 25 2020, 9:00 AM

Change 598421 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Test Update with actual producer/consumer stats

https://gerrit.wikimedia.org/r/598421

gerritbot added a comment.May 25 2020, 10:53 AM

Change 598422 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] Fix panic on prometheus.GaugeVec label cardinality mismatch

https://gerrit.wikimedia.org/r/598422

Stashbot added a comment.May 25 2020, 11:01 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-25T11:01:35Z] <ema> upload prometheus-rdkafka-exporter to buster-wikimedia T253197

Maintenance_bot removed a project: Patch-For-Review.May 25 2020, 11:10 AM
gerritbot added a comment.May 25 2020, 11:27 AM

Change 598444 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/atskafka@master] 0.7: use prometheus-rdkafka-exporter

https://gerrit.wikimedia.org/r/598444

gerritbot added a project: Patch-For-Review.May 25 2020, 11:27 AM
• ema mentioned this in T253551: atskafka: expose rdkafka metrics to prometheus .May 25 2020, 11:38 AM
• ema moved this task from Backlog to Caching on the Traffic board.
• ema closed this task as Resolved.May 25 2020, 12:17 PM
• ema claimed this task.

The package prometheus-rdkafka-exporter is now available in buster-wikimedia, closing.

gerritbot added a comment.May 25 2020, 12:18 PM

Change 598444 merged by Ema:
[operations/software/atskafka@master] 0.7: use prometheus-rdkafka-exporter

https://gerrit.wikimedia.org/r/598444

Maintenance_bot removed a project: Patch-For-Review.May 25 2020, 1:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL