reset of burrow metrics for consumer group
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hnowlan
	Jun 4 2020, 4:10 PM

Description

Due to a mistake during a deploy, we've ended up with some bad consumer group metrics in burrow for cpjobqueue. Would it be possible to get these reset?

The metrics in question in Prometheus can be seen at kafka_burrow_partition_lag{cluster="misc",exported_cluster="main-eqiad",group="cpjobqueue-low_traffic_jobs"}

Event Timeline

hnowlan created this task.Jun 4 2020, 4:10 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2020, 4:10 PM

hnowlan added a subscriber: Pchelolo.Jun 4 2020, 4:11 PM

Milimetric assigned this task to elukey.Jun 8 2020, 4:07 PM

Milimetric triaged this task as High priority.

Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.

elukey@kafkamon1001:~$ curl -X DELETE localhost:8100/v3/kafka/main-eqiad/consumer/cpjobqueue-low_traffic_jobs
{"error":false,"message":"consumer group removed","request":{"url":"/v3/kafka/main-eqiad/consumer/cpjobqueue-low_traffic_jobs","host":"kafkamon1001"}}

Mentioned in SAL (#wikimedia-operations) [2020-06-08T17:50:08Z] <elukey> restart prometheus burrow exporter for kafka main on kafkamon1001 - T254498

Seems good! https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?orgId=1&from=1591638119505&to=1591638865281&var-datasource=eqiad%20prometheus%2Fops&var-cluster=main-eqiad&var-topic=All&var-consumer_group=cpjobqueue-low_traffic_jobs

Great, thanks!

reset of burrow metrics for consumer groupClosed, ResolvedPublicActions

Description

Event Timeline

reset of burrow metrics for consumer group
Closed, ResolvedPublic
Actions