Page MenuHomePhabricator
Create Task
Maniphest T180442

Export burrow metrics to prometheus
Closed, ResolvedPublic13 Estimated Story Points
Actions

Description

Right now to calculate consumer lags in ChangeProp we use metrics supplied by librdkafka and try to subtract the committed offset from the hi offset per consumer group. However that doesn't really work - the metrics reported by librdkafka are quite unreliable. All these client-side metrics coming from librdkafka proved to be quite useless over the time we been reporting them so we might consider removing them.

However, we still want to know client offsets and we already have a special tool designed specifically for this purpose deployed - burrow. Let's also use https://github.com/jirwin/burrow_exporter to report the metrics to prometheus and then we'd be able to build some fancy graphs in grafana!

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
burrow: ensure pid file is created under /var/run/burrowElukeyoperations/puppetproduction+12 -2
burrow: fix zookeeper lock path valueElukeyoperations/puppetproduction+1 -1
role::kafka::monitoring: fix wrong cut in previous changeElukeyoperations/puppetproduction+5 -0
role::kafka::monitoring: add lag monitoring for JumboElukeyoperations/puppetproduction+14 -6
prometheus::ops|analytics: update Kafka Burrow's exporter configElukeyoperations/puppetproduction+2 -2
role::prometheus::ops: correct Kafka Burrow exporter's portElukeyoperations/puppetproduction+1 -1
role::prometheus::ops: remove broken Kafka burrow configElukeyoperations/puppetproduction+1 -6
role::prometheus::analytics|ops: add Kafka Burrow jobsElukeyoperations/puppetproduction+42 -2
profile::prometheus::burrow_exporter: set 0.0.0.0 as default listen addrElukeyoperations/puppetproduction+1 -1
profile::kafka::burrow: add prometheus monitoringElukeyoperations/puppetproduction+125 -5
burrow: parametrize configuration file namesElukeyoperations/puppetproduction+3 -3
role::webserver_misc_apps: add kafka burrow lag monitoring for mainElukeyoperations/puppetproduction+30 -6
burrow: add systemd supportElukeyoperations/puppetproduction+25 -3
role::kafka::analytics::burrow: move to profileElukeyoperations/puppetproduction+61 -45
Initial packagingElukeyoperations/debs/prometheus-burrow-exporterdebian+256 -0
Initial packaging of version 0.0.4Elukeyoperations/debs/prometheus-burrow-exportermaster+256 -0
package_builder: add dh-make-golang to the list of req. packagesElukeyoperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects

Event Timeline

• Pchelolo created this task.Nov 14 2017, 10:39 AM
Restricted Application added a project: Analytics. · View Herald TranscriptNov 14 2017, 10:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
elukey merged a task: T120852: Send burrow lag statistics to prometheus.Nov 14 2017, 10:49 AM
elukey added a project: User-Elukey.
elukey added subscribers: Ottomata, • madhuvishy, elukey.
Ottomata added a comment.Nov 14 2017, 2:00 PM

NICE! We've been putting off doing this for graphite for some time, so great that there is already a burrow exporter!

There is some limited configuration of burrow for the analytics Kafka cluster in puppet. We should make that more generic and set it up for all Kafka clusters by default.

fgiunchedi subscribed.Nov 16 2017, 4:16 PM
• fdans moved this task from Incoming to Wikistats on the Analytics board.Nov 16 2017, 4:29 PM
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.Nov 17 2017, 10:25 AM
fgiunchedi added a comment.Nov 17 2017, 12:54 PM

I took a quick look at burrow_exporter (without building/running it) and it looks sane. It seems to be polling burrow asynchronously every 30s by default and serve the metrics to prometheus when asked. Not a blocker but this is generally discouraged by prometheus best practices, there might be a good reason though (e.g. it is expensive to gather the metrics and thus some caching is needed)

• madhuvishy unsubscribed.Nov 27 2017, 6:58 PM
• fdans edited projects, added: Analytics-Kanban; removed: Analytics.Jan 18 2018, 6:30 PM
Ottomata removed Ottomata as the assignee of this task.Jan 22 2018, 4:06 PM
elukey claimed this task.Jan 22 2018, 4:07 PM
elukey added a comment.Jan 23 2018, 1:51 PM

I tried today to build the https://github.com/jirwin/burrow_exporter package with a simple Debian configuration on boron and I managed to get the deb built.

There are a couple of things that would need to be done though:

  1. Upgrade our Burrow version from 0.1 to 1.0 and deploy it to krypton.
  2. Ask upstream to merge/release a new version that supports v3 api (https://github.com/jirwin/burrow_exporter/issues/8).
  3. Build prometheus-burrow-exporter, configure puppet and deploy it.
elukey added a comment.Jan 23 2018, 2:39 PM

For burrow it turns out that the following list can replace the godeps in the debian dir:

Build-Depends: debhelper (>= 9), dh-golang, golang-go, dh-systemd (>= 1.5),
 golang-github-oneofone-xxhash-dev,
 golang-github-julienschmidt-httprouter-dev,
 golang-github-spf13-viper-dev,
 golang-go.uber-zap-dev,
 golang-gopkg-natefinch-lumberjack.v2-dev,
 golang-github-shopify-sarama-dev,
 golang-github-samuel-go-zookeeper-dev,
 golang-github-pborman-uuid-dev,

The main issue is the following:

src/github.com/linkedin/Burrow/core/internal/evaluator/caching.go:18:2: cannot find package "github.com/karrick/goswarm"

This package seems to be imported in sid, so we'd need to create it following: https://pkg-go.alioth.debian.org/packaging.html

elukey added a comment.Jan 23 2018, 3:26 PM
In T180442#3769470, @fgiunchedi wrote:

I took a quick look at burrow_exporter (without building/running it) and it looks sane. It seems to be polling burrow asynchronously every 30s by default and serve the metrics to prometheus when asked. Not a blocker but this is generally discouraged by prometheus best practices, there might be a good reason though (e.g. it is expensive to gather the metrics and thus some caching is needed)

I took a look to the code and it seems always doing the following:

  1. get all the kafka clusters
  2. for each cluster, get all the consumer groups
  3. for all the consumer groups, get metrics about lag etc..

If the number of consumer groups and partitions grows it might take a bit to process all of them, this might be the reason for the scrape interval (I don't see anything kept in memory between scrapes from a quick glance).

Ottomata added a comment.Jan 23 2018, 3:26 PM

If the number of consumer groups and partitions grows

It almost certainly will. :)

gerritbot subscribed.Jan 23 2018, 3:59 PM

Change 405892 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] package_builder: add dh-make-golang to the list of req. packages

https://gerrit.wikimedia.org/r/405892

gerritbot added a project: Patch-For-Review.Jan 23 2018, 3:59 PM
gerritbot added a comment.Jan 23 2018, 4:10 PM

Change 405892 merged by Elukey:
[operations/puppet@production] package_builder: add dh-make-golang to the list of req. packages

https://gerrit.wikimedia.org/r/405892

elukey added a comment.Feb 9 2018, 8:10 AM

It seems that recently https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=888673 was opened to maintain golang-github-karrick-goswarm and also Burrow is mentioned as ITP.

gerritbot added a comment.Feb 9 2018, 11:04 AM

Change 409308 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/debs/prometheus-burrow-exporter@master] Initial packaging of version 0.0.4

https://gerrit.wikimedia.org/r/409308

gerritbot added a comment.Feb 9 2018, 11:05 AM

Change 409308 abandoned by Elukey:
Initial packaging of version 0.0.4

https://gerrit.wikimedia.org/r/409308

gerritbot added a comment.Feb 9 2018, 11:15 AM

Change 409310 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/debs/prometheus-burrow-exporter@debian] Initial packaging

https://gerrit.wikimedia.org/r/409310

elukey added a comment.Feb 9 2018, 11:32 AM

Tested the package on krypton to have a good set of "real" metrics to check:

elukey@krypton:~$ curl localhost:8080/metrics -s  | grep -v "#" | sort
go_gc_duration_seconds_count 0
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_goroutines 14
go_memstats_alloc_bytes 1.589792e+06
go_memstats_alloc_bytes_total 1.589792e+06
go_memstats_buck_hash_sys_bytes 2812
go_memstats_frees_total 2735
go_memstats_gc_sys_bytes 169984
go_memstats_heap_alloc_bytes 1.589792e+06
go_memstats_heap_idle_bytes 466944
go_memstats_heap_inuse_bytes 2.220032e+06
go_memstats_heap_objects 20449
go_memstats_heap_released_bytes_total 0
go_memstats_heap_sys_bytes 2.686976e+06
go_memstats_last_gc_time_seconds 0
go_memstats_lookups_total 73
go_memstats_mallocs_total 23184
go_memstats_mcache_inuse_bytes 1736
go_memstats_mcache_sys_bytes 16384
go_memstats_mspan_inuse_bytes 34048
go_memstats_mspan_sys_bytes 49152
go_memstats_next_gc_bytes 4.473924e+06
go_memstats_other_sys_bytes 519428
go_memstats_stack_inuse_bytes 458752
go_memstats_stack_sys_bytes 458752
go_memstats_sys_bytes 3.903488e+06
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="0",topic="webrequest_text"} 1.95675232867e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="10",topic="webrequest_text"} 1.95666080007e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="11",topic="webrequest_text"} 1.95646227849e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="12",topic="webrequest_text"} 1.27570958161e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="13",topic="webrequest_text"} 1.27580268766e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="14",topic="webrequest_text"} 1.27571916404e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="15",topic="webrequest_text"} 1.27576537152e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="16",topic="webrequest_text"} 1.27563217801e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="17",topic="webrequest_text"} 1.27564689113e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="18",topic="webrequest_text"} 1.27554429657e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="19",topic="webrequest_text"} 1.27558182472e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="1",topic="webrequest_text"} 1.95659361433e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="20",topic="webrequest_text"} 1.27561856202e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="21",topic="webrequest_text"} 1.27561933715e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="22",topic="webrequest_text"} 1.27560583951e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="23",topic="webrequest_text"} 1.27556410983e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="2",topic="webrequest_text"} 1.95692242721e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="3",topic="webrequest_text"} 1.95655427546e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="4",topic="webrequest_text"} 1.95668001133e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="5",topic="webrequest_text"} 1.95653111785e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="6",topic="webrequest_text"} 1.95695962447e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="7",topic="webrequest_text"} 1.95597743904e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="8",topic="webrequest_text"} 1.95672359543e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="9",topic="webrequest_text"} 1.95658909524e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="0",topic="eventlogging-client-side"} 9.591305e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="0",topic="eventlogging-valid-mixed"} 7.42751883e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="10",topic="eventlogging-client-side"} 8.33979306e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="10",topic="eventlogging-valid-mixed"} 7.38117896e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="11",topic="eventlogging-client-side"} 8.34489515e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="11",topic="eventlogging-valid-mixed"} 7.42293043e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="1",topic="eventlogging-client-side"} 8.34188519e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="1",topic="eventlogging-valid-mixed"} 7.42778564e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="2",topic="eventlogging-client-side"} 8.34612357e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="2",topic="eventlogging-valid-mixed"} 7.42820254e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="3",topic="eventlogging-client-side"} 8.34496e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="3",topic="eventlogging-valid-mixed"} 7.41086364e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="4",topic="eventlogging-client-side"} 8.34556701e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="4",topic="eventlogging-valid-mixed"} 7.38069277e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="5",topic="eventlogging-client-side"} 8.33638028e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="5",topic="eventlogging-valid-mixed"} 7.42277278e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="6",topic="eventlogging-client-side"} 8.34264953e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="6",topic="eventlogging-valid-mixed"} 7.42797142e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="7",topic="eventlogging-client-side"} 8.34696719e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="7",topic="eventlogging-valid-mixed"} 7.42802942e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="8",topic="eventlogging-client-side"} 8.3470121e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="8",topic="eventlogging-valid-mixed"} 7.42794243e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="9",topic="eventlogging-client-side"} 8.33594367e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="9",topic="eventlogging-valid-mixed"} 7.4000729e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="0",topic="eventlogging-valid-mixed"} 7.42751929e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="10",topic="eventlogging-valid-mixed"} 7.38117938e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="11",topic="eventlogging-valid-mixed"} 7.42293084e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="1",topic="eventlogging-valid-mixed"} 7.42778597e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="2",topic="eventlogging-valid-mixed"} 7.42820281e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="3",topic="eventlogging-valid-mixed"} 7.41086394e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="4",topic="eventlogging-valid-mixed"} 7.38069316e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="5",topic="eventlogging-valid-mixed"} 7.42277324e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="6",topic="eventlogging-valid-mixed"} 7.42797179e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="7",topic="eventlogging-valid-mixed"} 7.42802972e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="8",topic="eventlogging-valid-mixed"} 7.42794281e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="9",topic="eventlogging-valid-mixed"} 7.40007326e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-create"} 912
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-delete"} 153120
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-move"} 72422
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-undelete"} 2405
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-create"} 4.3216579e+07
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-delete"} 5.129216e+06
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-move"} 2.698294e+06
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-undelete"} 100519
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="0",topic="eventlogging-client-side"} 9.59130511e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="10",topic="eventlogging-client-side"} 8.33979309e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="11",topic="eventlogging-client-side"} 8.34489515e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="1",topic="eventlogging-client-side"} 8.34188519e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="2",topic="eventlogging-client-side"} 8.34612366e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="3",topic="eventlogging-client-side"} 8.34496011e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="4",topic="eventlogging-client-side"} 8.34556708e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="5",topic="eventlogging-client-side"} 8.33638029e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="6",topic="eventlogging-client-side"} 8.34264953e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="7",topic="eventlogging-client-side"} 8.34696849e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="8",topic="eventlogging-client-side"} 8.34701212e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="9",topic="eventlogging-client-side"} 8.33594374e+08
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="0",topic="webrequest_text"} 1.95675208455e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="10",topic="webrequest_text"} 1.95666054715e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="11",topic="webrequest_text"} 1.95646204972e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="12",topic="webrequest_text"} 1.27570933798e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="13",topic="webrequest_text"} 1.27580244536e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="14",topic="webrequest_text"} 1.2757189359e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="15",topic="webrequest_text"} 1.27576514517e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="16",topic="webrequest_text"} 1.2756319285e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="17",topic="webrequest_text"} 1.27564664677e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="18",topic="webrequest_text"} 1.27554405568e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="19",topic="webrequest_text"} 1.27558159564e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="1",topic="webrequest_text"} 1.95659336357e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="20",topic="webrequest_text"} 1.27561833134e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="21",topic="webrequest_text"} 1.27561908709e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="22",topic="webrequest_text"} 1.27560558904e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="23",topic="webrequest_text"} 1.27556386835e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="2",topic="webrequest_text"} 1.95692218272e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="3",topic="webrequest_text"} 1.95655403432e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="4",topic="webrequest_text"} 1.95667976337e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="5",topic="webrequest_text"} 1.95653088838e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="6",topic="webrequest_text"} 1.9569593805e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="7",topic="webrequest_text"} 1.95597719371e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="8",topic="webrequest_text"} 1.95672336904e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="9",topic="webrequest_text"} 1.95658885333e+11
kafka_burrow_partition_current_offset{cluster="eqiad",group="mjolnir",partition="0",topic="mjolnir_request"} 226469
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="0",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="10",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="11",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="12",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="13",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="14",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="15",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="16",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="17",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="18",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="19",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="1",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="20",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="21",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="22",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="23",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="2",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="3",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="4",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="5",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="6",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="7",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="8",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="9",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="0",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="0",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="10",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="10",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="11",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="11",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="1",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="1",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="2",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="2",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="3",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="3",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="4",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="4",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="5",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="5",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="6",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="6",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="7",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="7",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="8",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="8",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="9",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_files_00",partition="9",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="0",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="10",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="11",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="1",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="2",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="3",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="4",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="5",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="6",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="7",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="8",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="9",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-create"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-delete"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-move"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-undelete"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-create"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-delete"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-move"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-undelete"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="0",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="10",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="11",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="1",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="2",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="3",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="4",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="5",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="6",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="7",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="8",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="9",topic="eventlogging-client-side"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="0",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="10",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="11",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="12",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="13",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="14",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="15",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="16",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="17",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="18",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="19",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="1",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="20",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="21",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="22",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="23",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="2",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="3",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="4",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="5",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="6",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="7",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="8",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="frack_kafkatee_alnitak",partition="9",topic="webrequest_text"} 0
kafka_burrow_partition_lag{cluster="eqiad",group="mjolnir",partition="0",topic="mjolnir_request"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="0",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="10",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="11",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="12",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="13",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="14",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="15",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="16",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="17",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="18",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="19",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="1",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="20",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="21",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="22",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="23",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="2",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="3",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="4",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="5",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="6",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="7",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="8",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="americium.frack.eqiad.wmnet",partition="9",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="0",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="0",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="10",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="10",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="11",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="11",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="1",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="1",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="2",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="2",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="3",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="3",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="4",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="4",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="5",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="5",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="6",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="6",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="7",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="7",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="8",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="8",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="9",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_files_00",partition="9",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="0",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="10",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="11",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="1",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="2",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="3",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="4",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="5",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="6",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="7",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="8",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_00",partition="9",topic="eventlogging-valid-mixed"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-create"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-delete"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-move"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="codfw.mediawiki.page-undelete"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-create"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-delete"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-move"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00",partition="0",topic="eqiad.mediawiki.page-undelete"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="0",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="10",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="11",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="1",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="2",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="3",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="4",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="5",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="6",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="7",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="8",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="eventlogging_processor_client_side_00",partition="9",topic="eventlogging-client-side"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="0",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="10",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="11",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="12",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="13",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="14",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="15",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="16",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="17",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="18",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="19",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="1",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="20",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="21",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="22",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="23",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="2",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="3",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="4",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="5",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="6",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="7",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="8",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="frack_kafkatee_alnitak",partition="9",topic="webrequest_text"} 0
kafka_burrow_partition_max_offset{cluster="eqiad",group="mjolnir",partition="0",topic="mjolnir_request"} 0
kafka_burrow_total_lag{cluster="eqiad",group="americium.frack.eqiad.wmnet"} 0
kafka_burrow_total_lag{cluster="eqiad",group="eventlogging_consumer_files_00"} 0
kafka_burrow_total_lag{cluster="eqiad",group="eventlogging_consumer_mysql_00"} 0
kafka_burrow_total_lag{cluster="eqiad",group="eventlogging_consumer_mysql_eventbus_00"} 0
kafka_burrow_total_lag{cluster="eqiad",group="eventlogging_processor_client_side_00"} 0
kafka_burrow_total_lag{cluster="eqiad",group="frack_kafkatee_alnitak"} 0
kafka_burrow_total_lag{cluster="eqiad",group="mjolnir"} 0
process_cpu_seconds_total 0.01
process_max_fds 1024
process_open_fds 9
process_resident_memory_bytes 8.25344e+06
process_start_time_seconds 1.51817585517e+09
process_virtual_memory_bytes 2.68951552e+08
elukey added a comment.EditedFeb 9 2018, 11:39 AM

Main pain points:

  • on krypton we are using Burrow version 0.0.1, meanwhile upstream is already to 1.0. Packaging 1.0 requires a golang package still not in Sid (but ITP https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=888673) that could be easily created though and uploaded to our apt repos. Burrow itself seems to be mentioned as future ITP in Sid.
  • Burrow 1.0 offers new breaking APIs (v3) to inspect its metrics, Burrow 0.0.1 of course uses the previous version.

Giving this situation I'd be inclined to just keep Burrow 0.0.1 for the moment and add the support for the current version of the burrow exporter, since it will be useful for the Services team to monitor their metrics (we still need to add support/config for Kafka main to Burrow though). We'll upgrade as soon as possible, dealing with the metrics that will change (even if after reading the gh issue I don't expect a lot of troubles).

@fgiunchedi/@Pchelolo - thoughts?

elukey moved this task from Analytics Backlog to In Progress on the User-Elukey board.Feb 9 2018, 2:21 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Feb 9 2018, 3:19 PM
fgiunchedi added a comment.Feb 9 2018, 4:07 PM
In T180442#3957980, @elukey wrote:

Main pain points:

  • on krypton we are using Burrow version 0.0.1, meanwhile upstream is already to 1.0. Packaging 1.0 requires a golang package still not in Sid (but ITP https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=888673) that could be easily created though and uploaded to our apt repos. Burrow itself seems to be mentioned as future ITP in Sid.
  • Burrow 1.0 offers new breaking APIs (v3) to inspect its metrics, Burrow 0.0.1 of course uses the previous version.

Giving this situation I'd be inclined to just keep Burrow 0.0.1 for the moment and add the support for the current version of the burrow exporter, since it will be useful for the Services team to monitor their metrics (we still need to add support/config for Kafka main to Burrow though). We'll upgrade as soon as possible, dealing with the metrics that will change (even if after reading the gh issue I don't expect a lot of troubles).

@fgiunchedi/@Pchelolo - thoughts?

Your plan sounds good to me, let's go ahead with what we have now and we'll deal with the Burrow 1.0 upgrade when the time comes.

gerritbot added a comment.Feb 12 2018, 11:01 AM

Change 409310 merged by Elukey:
[operations/debs/prometheus-burrow-exporter@debian] Initial packaging

https://gerrit.wikimedia.org/r/409310

gerritbot added a comment.Feb 12 2018, 3:21 PM

Change 409912 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::kafka::analytics::burrow: move to profile

https://gerrit.wikimedia.org/r/409912

gerritbot added a comment.Feb 13 2018, 2:55 PM

Change 409912 merged by Elukey:
[operations/puppet@production] role::kafka::analytics::burrow: move to profile

https://gerrit.wikimedia.org/r/409912

gerritbot added a comment.Feb 13 2018, 3:23 PM

Change 410186 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] burrow: add systemd support

https://gerrit.wikimedia.org/r/410186

gerritbot added a comment.Feb 13 2018, 3:35 PM

Change 410186 merged by Elukey:
[operations/puppet@production] burrow: add systemd support

https://gerrit.wikimedia.org/r/410186

gerritbot added a comment.Feb 13 2018, 5:25 PM

Change 410212 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::webserver_misc_apps: add kafka burrow lag monitoring for main

https://gerrit.wikimedia.org/r/410212

gerritbot added a comment.Feb 13 2018, 5:41 PM

Change 410212 merged by Elukey:
[operations/puppet@production] role::webserver_misc_apps: add kafka burrow lag monitoring for main

https://gerrit.wikimedia.org/r/410212

gerritbot added a comment.Feb 13 2018, 5:48 PM

Change 410219 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] burrow: parametrize configuration file names

https://gerrit.wikimedia.org/r/410219

gerritbot added a comment.Feb 13 2018, 6:00 PM

Change 410219 merged by Elukey:
[operations/puppet@production] burrow: parametrize configuration file names

https://gerrit.wikimedia.org/r/410219

elukey added a comment.Feb 13 2018, 6:12 PM
elukey@krypton:~$ curl http://localhost:8100/v2/kafka/main-eqiad/consumer
{"error":false,"message":"consumer list returned","consumers":["change-prop-purge_varnish-resource_change","change-prop-ores_cache_2","change-prop-on_transclusion_update","change-prop-on_wikidata_description_change","change-prop-mobile_rerender-resource_change","change-prop-updateBetaFeaturesUserCounts","change-prop-wikibase-addUsagesForPage","change-prop-on_backlinks_update","change-prop-page_images_mobile","change-prop-page_restore","change-prop-revision_visibility_change","change-prop-RecordLintJob","change-prop-summary_definition_rerender-resource_change","change-prop-flaggedrevs_CacheUpdate","change-prop-page_images_summary","change-prop-ores_cache_1","change-prop-htmlCacheUpdate","change-prop-mw_purge","change-prop-page_move","change-prop-page_edit","change-prop-wikidata_description_on_edit","change-prop-page_delete","change-prop-null_edit","change-prop-page_create"]}

elukey@krypton:~$ curl http://localhost:8200/v2/kafka/main-codfw/consumer
{"error":false,"message":"consumer list returned","consumers":["change-prop-summary_definition_rerender-change-prop_transcludes_resource-change","change-prop-summary_definition_rerender-resource_change","change-prop-wikibase-addUsagesForPage","change-prop-purge_varnish-change-prop_transcludes_resource-change","change-prop-mobile_rerender-change-prop_transcludes_resource-change","change-prop-mobile_rerender-resource_change","change-prop-purge_varnish-resource_change"]}
elukey@krypton:~$ sudo systemctl status burrow*
● burrow-main-codfw.service - Burrow - Kafka main-codfw Consumer Offset Monitor
   Loaded: loaded (/lib/systemd/system/burrow-main-codfw.service; enabled)
   Active: active (running) since Tue 2018-02-13 18:04:13 UTC; 7min ago
 Main PID: 6870 (Burrow)
   CGroup: /system.slice/burrow-main-codfw.service
           └─6870 /usr/bin/Burrow --config /etc/burrow/burrow-main-codfw.cfg

Feb 13 18:04:13 krypton systemd[1]: Started Burrow - Kafka main-codfw Consumer Offset Monitor.
Feb 13 18:04:13 krypton Burrow[6870]: Reading configuration from /etc/burrow/burrow-main-codfw.cfg

● burrow.service - Burrow - Kafka Consumer Offset Monitor
   Loaded: loaded (/lib/systemd/system/burrow.service; enabled)
   Active: inactive (dead) since Tue 2018-02-13 15:36:37 UTC; 2h 34min ago
 Main PID: 478 (code=exited, status=0/SUCCESS)

Feb 13 15:36:37 krypton systemd[1]: Stopping Burrow - Kafka Consumer Offset Monitor...
Feb 13 15:36:37 krypton systemd[1]: Stopped Burrow - Kafka Consumer Offset Monitor.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

● burrow-main-eqiad.service - Burrow - Kafka main-eqiad Consumer Offset Monitor
   Loaded: loaded (/lib/systemd/system/burrow-main-eqiad.service; enabled)
   Active: active (running) since Tue 2018-02-13 18:04:13 UTC; 7min ago
 Main PID: 6812 (Burrow)
   CGroup: /system.slice/burrow-main-eqiad.service
           └─6812 /usr/bin/Burrow --config /etc/burrow/burrow-main-eqiad.cfg

Feb 13 18:04:13 krypton systemd[1]: Started Burrow - Kafka main-eqiad Consumer Offset Monitor.
Feb 13 18:04:13 krypton Burrow[6812]: Reading configuration from /etc/burrow/burrow-main-eqiad.cfg

● burrow-eqiad.service - Burrow - Kafka eqiad Consumer Offset Monitor
   Loaded: loaded (/lib/systemd/system/burrow-eqiad.service; enabled)
   Active: active (running) since Tue 2018-02-13 18:04:13 UTC; 7min ago
 Main PID: 6755 (Burrow)
   CGroup: /system.slice/burrow-eqiad.service
           └─6755 /usr/bin/Burrow --config /etc/burrow/burrow-eqiad.cfg

Feb 13 18:04:13 krypton systemd[1]: Started Burrow - Kafka eqiad Consumer Offset Monitor.
Feb 13 18:04:13 krypton Burrow[6755]: Reading configuration from /etc/burrow/burrow-eqiad.cfg

(note: burrow.service is the basic unit that must be stopped/disabled)

It seems working fine!

gerritbot added a comment.Feb 16 2018, 3:03 PM

Change 411249 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::kafka::burrow: add prometheus monitoring

https://gerrit.wikimedia.org/r/411249

gerritbot added a comment.Feb 19 2018, 2:08 PM

Change 411249 merged by Elukey:
[operations/puppet@production] profile::kafka::burrow: add prometheus monitoring

https://gerrit.wikimedia.org/r/411249

gerritbot added a comment.Feb 19 2018, 2:20 PM

Change 412709 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::prometheus::burrow_exporter: set 0.0.0.0 as default listen addr

https://gerrit.wikimedia.org/r/412709

gerritbot added a comment.Feb 19 2018, 2:20 PM

Change 412709 merged by Elukey:
[operations/puppet@production] profile::prometheus::burrow_exporter: set 0.0.0.0 as default listen addr

https://gerrit.wikimedia.org/r/412709

gerritbot added a comment.Feb 20 2018, 11:36 AM

Change 412881 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::prometheus::analytics|ops: add Kafka Burrow jobs

https://gerrit.wikimedia.org/r/412881

gerritbot added a comment.Feb 20 2018, 2:15 PM

Change 412881 merged by Elukey:
[operations/puppet@production] role::prometheus::analytics|ops: add Kafka Burrow jobs

https://gerrit.wikimedia.org/r/412881

elukey mentioned this in T187805: Ganeti instances to support Kafka Burrow Consumer lag monitoring.Feb 20 2018, 4:47 PM

So summary:

gerritbot added a comment.Feb 20 2018, 5:23 PM

Change 412959 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::prometheus::ops: remove broken Kafka burrow config

https://gerrit.wikimedia.org/r/412959

gerritbot added a comment.Feb 20 2018, 5:24 PM

Change 412959 merged by Elukey:
[operations/puppet@production] role::prometheus::ops: remove broken Kafka burrow config

https://gerrit.wikimedia.org/r/412959

elukey moved this task from In Progress to Stalled on the User-Elukey board.Feb 20 2018, 5:44 PM
gerritbot added a comment.Feb 21 2018, 8:01 AM

Change 413107 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::prometheus::ops: correct Kafka Burrow exporter's port

https://gerrit.wikimedia.org/r/413107

gerritbot added a comment.Feb 21 2018, 8:02 AM

Change 413107 merged by Elukey:
[operations/puppet@production] role::prometheus::ops: correct Kafka Burrow exporter's port

https://gerrit.wikimedia.org/r/413107

gerritbot added a comment.Feb 26 2018, 9:32 AM

Change 414632 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] prometheus::ops|analytics: update Kafka Burrow's exporter config

https://gerrit.wikimedia.org/r/414632

gerritbot added a comment.Feb 26 2018, 9:34 AM

Change 414632 merged by Elukey:
[operations/puppet@production] prometheus::ops|analytics: update Kafka Burrow's exporter config

https://gerrit.wikimedia.org/r/414632

gerritbot added a comment.Feb 26 2018, 9:49 AM

Change 414636 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::kafka::monitoring: add lag monitoring for Jumbo

https://gerrit.wikimedia.org/r/414636

gerritbot added a comment.Feb 26 2018, 9:52 AM

Change 414636 merged by Elukey:
[operations/puppet@production] role::kafka::monitoring: add lag monitoring for Jumbo

https://gerrit.wikimedia.org/r/414636

gerritbot added a comment.Feb 26 2018, 10:31 AM

Change 414639 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::kafka::monitoring: fix wrong cut in previous change

https://gerrit.wikimedia.org/r/414639

gerritbot added a comment.Feb 26 2018, 10:32 AM

Change 414639 merged by Elukey:
[operations/puppet@production] role::kafka::monitoring: fix wrong cut in previous change

https://gerrit.wikimedia.org/r/414639

elukey added a comment.Feb 26 2018, 1:48 PM

We have now main-eqiad/codfw and analytics monitored, but jumbo is not supported yet since as far as I can see Burrow 0.1 doesn't support fetching consumer status from the __consumer* topics (and this makes sense since 0.1 is super old). Burrow 1.0 seems to export v2 and v3's APIs, so it might be compatible with the current version of the Prometheus Burrow exporter, I'll need to investigate and test.

elukey moved this task from Stalled to In Progress on the User-Elukey board.Feb 27 2018, 1:02 PM
gerritbot added a comment.Mar 2 2018, 9:51 AM

Change 415818 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] burrow: fix zookeeper lock path value

https://gerrit.wikimedia.org/r/415818

gerritbot added a comment.Mar 2 2018, 9:52 AM

Change 415818 merged by Elukey:
[operations/puppet@production] burrow: fix zookeeper lock path value

https://gerrit.wikimedia.org/r/415818

elukey added a comment.Mar 2 2018, 10:04 AM

I managed to build Burrow 1.0 and I've tested it on kafkamon1001, but so far these are the issues:

  1. confirmed that the /v2/ API seems gone, so the prometheus-burrow-exporter will not work if upstream doesn't update it.
  1. I failed to get the list of consumer groups for analytics and jumbo eqiad, need more testing.

The current issue for analytics is that we'd like to migrate the eventlogging topic producers (varnishkafka) to Jumbo, but in this way we'd loose Burrow monitoring (since 0.1 does not support recent kafka __consumer_offset topics).

elukey added a comment.Mar 2 2018, 11:23 AM

All right I proved to myself today that I am definitely stupid. I saw the following today:

https://grafana.wikimedia.org/dashboard/db/kafka-consumer-lag?orgId=1&from=now-24h&to=now&var-datasource=eqiad%20prometheus%2Fops&var-cluster=jumbo-eqiad&var-topic=All&var-consumer_group=80ff0fe8-27ef-4738-92ab-c5738598fa54

And I was wondering how it was possible. This is a consequence of https://phabricator.wikimedia.org/T188252#4015386, namely a new Jumbo consumer being activated. So Burrow 0.1.0 works with Jumbo too, and 1.0's migration can be postponed.

In my head, for some reason, I confused the old vs new Kafka consumer support (zookeeper vs consumer_offset) with Kafka 0.9 vs 1.0. The consumer_offset topics are available for 0.9 too, and Burrow 0.1.0 can read them without any issue.

So I am inclined to close this task and open a new one to migrate to Burrow 1.0 when the prometheus exporter will be updated!

elukey set the point value for this task to 13.Mar 2 2018, 11:23 AM
elukey moved this task from In Progress to Done on the Analytics-Kanban board.
elukey mentioned this in T188719: Upgrade Kafka Burrow to 1.0.Mar 2 2018, 11:36 AM
elukey moved this task from In Progress to Done on the User-Elukey board.Mar 2 2018, 11:39 AM
gerritbot added a comment.Mar 3 2018, 3:39 PM

Change 416232 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] burrow: ensure pid file is created under /var/run/burrow

https://gerrit.wikimedia.org/r/416232

gerritbot added a comment.Mar 3 2018, 3:50 PM

Change 416232 merged by Elukey:
[operations/puppet@production] burrow: ensure pid file is created under /var/run/burrow

https://gerrit.wikimedia.org/r/416232

• Nuria closed this task as Resolved.Mar 26 2018, 9:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits