Page MenuHomePhabricator

reset of burrow metrics for consumer group
Closed, ResolvedPublic

Description

Due to a mistake during a deploy, we've ended up with some bad consumer group metrics in burrow for cpjobqueue. Would it be possible to get these reset?

The metrics in question in Prometheus can be seen at kafka_burrow_partition_lag{cluster="misc",exported_cluster="main-eqiad",group="cpjobqueue-low_traffic_jobs"}

Event Timeline

hnowlan created this task.Jun 4 2020, 4:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2020, 4:10 PM
Milimetric assigned this task to elukey.Jun 8 2020, 4:07 PM
Milimetric triaged this task as High priority.
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.
elukey added a comment.Jun 8 2020, 5:49 PM
elukey@kafkamon1001:~$ curl -X DELETE localhost:8100/v3/kafka/main-eqiad/consumer/cpjobqueue-low_traffic_jobs
{"error":false,"message":"consumer group removed","request":{"url":"/v3/kafka/main-eqiad/consumer/cpjobqueue-low_traffic_jobs","host":"kafkamon1001"}}

Mentioned in SAL (#wikimedia-operations) [2020-06-08T17:50:08Z] <elukey> restart prometheus burrow exporter for kafka main on kafkamon1001 - T254498

Great, thanks!