Page MenuHomePhabricator
Create Task
Maniphest T253551

atskafka: expose rdkafka metrics to prometheus
Closed, ResolvedPublic
Actions

Description

By using prometheus-rdkafka-exporter (T253197) we should make atskafka expose kafka prometheus metrics natively. The Prometheus servers should fetch the metrics made available by atskafka.

Details

SubjectRepoBranchLines +/-
prometheus: whitelist atskafka runtime metricsoperations/puppetproduction+1 -1
0.9: build against rdkafka-exporter 0.2operations/software/atskafkamaster+11 -11
0.2: pass the appropriate list of labels for the metricoperations/software/prometheus-rdkafka-exportermaster+25 -3
atskafka: remove trailing whitespace from queue.buffering.max.msoperations/puppetproduction+1 -1
atskafka: increase queue.buffering.max.msoperations/puppetproduction+9 -0
prometheus: whitelist atskafka-specific metricsoperations/puppetproduction+1 -1
0.8: add new prometheus metricsoperations/software/atskafkamaster+28 -1
prometheus: job definition for atskafkaoperations/puppetproduction+27 -1
atskafka: do not write stats to diskoperations/puppetproduction+1 -12
0.7: use prometheus-rdkafka-exporteroperations/software/atskafkamaster+25 -20
Customize query in gerrit

Related Objects

Event Timeline

• ema created this task.May 25 2020, 11:38 AM
Restricted Application added a project: SRE. · View Herald TranscriptMay 25 2020, 11:38 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• ema triaged this task as Medium priority.May 25 2020, 11:39 AM
• ema moved this task from Backlog to Caching on the Traffic board.
gerritbot added a comment.May 25 2020, 11:39 AM

Change 598444 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/atskafka@master] 0.7: use prometheus-rdkafka-exporter

https://gerrit.wikimedia.org/r/598444

gerritbot added a project: Patch-For-Review.May 25 2020, 11:39 AM
gerritbot added a comment.May 25 2020, 12:17 PM

Change 598454 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] atskafka: do not write stats to disk

https://gerrit.wikimedia.org/r/598454

gerritbot added a comment.May 25 2020, 12:18 PM

Change 598444 merged by Ema:
[operations/software/atskafka@master] 0.7: use prometheus-rdkafka-exporter

https://gerrit.wikimedia.org/r/598444

gerritbot added a comment.May 25 2020, 12:34 PM

Change 598454 merged by Ema:
[operations/puppet@production] atskafka: do not write stats to disk

https://gerrit.wikimedia.org/r/598454

Stashbot added a comment.May 25 2020, 12:44 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-25T12:44:38Z] <ema> upload atskafka 0.7 to buster-wikimedia, upgrade cp3050 T253551

gerritbot added a comment.May 25 2020, 12:49 PM

Change 598464 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: job definition for atskafka

https://gerrit.wikimedia.org/r/598464

gerritbot added a comment.May 27 2020, 10:42 AM

Change 598464 merged by Ema:
[operations/puppet@production] prometheus: job definition for atskafka

https://gerrit.wikimedia.org/r/598464

Maintenance_bot removed a project: Patch-For-Review.May 27 2020, 11:10 AM
gerritbot added a comment.May 28 2020, 1:05 PM

Change 599322 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/atskafka@master] 0.8: add new prometheus metrics

https://gerrit.wikimedia.org/r/599322

gerritbot added a project: Patch-For-Review.May 28 2020, 1:05 PM
gerritbot added a comment.May 28 2020, 1:49 PM

Change 599322 merged by Ema:
[operations/software/atskafka@master] 0.8: add new prometheus metrics

https://gerrit.wikimedia.org/r/599322

Stashbot added a comment.May 28 2020, 2:01 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-28T14:01:03Z] <ema> atskafka 0.8 uploaded to buster-wikimedia T253551

gerritbot added a comment.May 28 2020, 2:05 PM

Change 599341 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: whitelist atskafka-specific metrics

https://gerrit.wikimedia.org/r/599341

gerritbot added a comment.May 28 2020, 2:12 PM

Change 599341 merged by Ema:
[operations/puppet@production] prometheus: whitelist atskafka-specific metrics

https://gerrit.wikimedia.org/r/599341

Maintenance_bot removed a project: Patch-For-Review.May 28 2020, 3:10 PM
gerritbot added a comment.May 29 2020, 7:32 AM

Change 599685 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] atskafka: increase queue.buffering.max.ms

https://gerrit.wikimedia.org/r/599685

gerritbot added a project: Patch-For-Review.May 29 2020, 7:32 AM
gerritbot added a comment.May 29 2020, 7:47 AM

Change 599685 merged by Ema:
[operations/puppet@production] atskafka: increase queue.buffering.max.ms

https://gerrit.wikimedia.org/r/599685

gerritbot added a comment.May 29 2020, 8:01 AM

Change 599697 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] atskafka: remove trailing whitespace from queue.buffering.max.ms

https://gerrit.wikimedia.org/r/599697

gerritbot added a comment.May 29 2020, 8:03 AM

Change 599697 merged by Ema:
[operations/puppet@production] atskafka: remove trailing whitespace from queue.buffering.max.ms

https://gerrit.wikimedia.org/r/599697

Maintenance_bot removed a project: Patch-For-Review.May 29 2020, 8:10 AM
gerritbot added a comment.May 29 2020, 9:32 AM

Change 599759 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/prometheus-rdkafka-exporter@master] 0.2: pass the appropriate list of labels for the metric

https://gerrit.wikimedia.org/r/599759

gerritbot added a project: Patch-For-Review.May 29 2020, 9:32 AM
gerritbot added a comment.May 29 2020, 10:53 AM

Change 599759 merged by Ema:
[operations/software/prometheus-rdkafka-exporter@master] 0.2: pass the appropriate list of labels for the metric

https://gerrit.wikimedia.org/r/599759

Stashbot added a comment.May 29 2020, 11:01 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-29T11:01:12Z] <ema> upload prometheus-rdkafka-exporter 0.2 to buster-wikimedia T253551

gerritbot added a comment.May 29 2020, 11:01 AM

Change 599786 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/atskafka@master] 0.9: build against rdkafka-exporter 0.2

https://gerrit.wikimedia.org/r/599786

gerritbot added a comment.May 29 2020, 11:12 AM

Change 599786 merged by Ema:
[operations/software/atskafka@master] 0.9: build against rdkafka-exporter 0.2

https://gerrit.wikimedia.org/r/599786

Maintenance_bot removed a project: Patch-For-Review.May 29 2020, 12:10 PM
• ema closed this task as Resolved.Jun 2 2020, 8:13 AM
• ema claimed this task.

This is now done: atskafka uses prometheus-rdkafka-exporter as a library and exposes rdkafka statistics to prometheus natively, together with its own metrics such as atskafka_delivery_errors_total and atskafka_seq_number.

gerritbot added a comment.Jul 27 2020, 7:34 AM

Change 616444 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: whitelist atskafka runtime metrics

https://gerrit.wikimedia.org/r/616444

gerritbot added a project: Patch-For-Review.Jul 27 2020, 7:34 AM
gerritbot added a comment.Jul 27 2020, 10:51 AM

Change 616444 merged by Ema:
[operations/puppet@production] prometheus: whitelist atskafka runtime metrics

https://gerrit.wikimedia.org/r/616444

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL