Page MenuHomePhabricator
Create Task
Maniphest T189611

Alert for Kafka MirrorMaker lag
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

We should know if MirrorMaker instances are lagging. We do track lag in graphite, so we could use icinga + graphite for this, or we could use Burrow.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.Mar 13 2018, 6:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2018, 6:14 PM
Ottomata triaged this task as Medium priority.Mar 13 2018, 6:14 PM
Ottomata added projects: Analytics, Analytics-Clusters.
Ottomata set the point value for this task to 5.
Ottomata added a parent task: T189464: Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo.
Ottomata moved this task from Incoming to Kafka Work on the Analytics board.Mar 15 2018, 4:35 PM
Ottomata added a project: Analytics-Kanban.
fdans moved this task from Next Up to Paused on the Analytics-Kanban board.Mar 22 2018, 4:46 PM
Ottomata moved this task from Paused to In Progress on the Analytics-Kanban board.Mar 27 2018, 2:51 PM
gerritbot added a subscriber: gerritbot.Mar 27 2018, 2:52 PM

Change 422163 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Install nrpe check for Kafka consumer lag by checking burrow

https://gerrit.wikimedia.org/r/422163

gerritbot added a project: Patch-For-Review.Mar 27 2018, 2:52 PM
gerritbot added a comment.Mar 27 2018, 5:09 PM

Change 422163 merged by Ottomata:
[operations/puppet@production] Install nrpe check for Kafka consumer lag by checking burrow

https://gerrit.wikimedia.org/r/422163

gerritbot added a comment.Mar 27 2018, 5:41 PM

Change 422192 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add mirror_name and host as labels for mirror maker prometheus

https://gerrit.wikimedia.org/r/422192

gerritbot added a comment.Mar 27 2018, 5:48 PM

Change 422192 merged by Ottomata:
[operations/puppet@production] Add mirror_name and host as labels for mirror maker prometheus

https://gerrit.wikimedia.org/r/422192

gerritbot added a comment.Mar 27 2018, 6:08 PM

Change 422201 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Can't set labels on metric without name set

https://gerrit.wikimedia.org/r/422201

gerritbot added a comment.Mar 27 2018, 6:08 PM

Change 422201 merged by Ottomata:
[operations/puppet@production] Can't set labels on metric without name set

https://gerrit.wikimedia.org/r/422201

gerritbot added a comment.Mar 27 2018, 7:13 PM

Change 422230 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add check_prometheus alerts for Kafka MirrorMaker instances.

https://gerrit.wikimedia.org/r/422230

gerritbot added a comment.Mar 27 2018, 7:33 PM

Change 422230 merged by Ottomata:
[operations/puppet@production] Add check_prometheus alerts for Kafka MirrorMaker instances.

https://gerrit.wikimedia.org/r/422230

gerritbot added a comment.Mar 27 2018, 7:40 PM

Change 422251 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix path to check_kafka_consumer_lag nrpe check

https://gerrit.wikimedia.org/r/422251

gerritbot added a comment.Mar 27 2018, 7:43 PM

Change 422251 merged by Ottomata:
[operations/puppet@production] Fix path to check_kafka_consumer_lag nrpe check

https://gerrit.wikimedia.org/r/422251

gerritbot added a comment.Mar 27 2018, 7:57 PM

Change 422258 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix prometheus_url for mirror maker alert

https://gerrit.wikimedia.org/r/422258

gerritbot added a comment.Mar 27 2018, 7:58 PM

Change 422258 merged by Ottomata:
[operations/puppet@production] Fix prometheus_url for mirror maker alert

https://gerrit.wikimedia.org/r/422258

gerritbot added a comment.Mar 27 2018, 8:17 PM

Change 422335 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use scalar in dropped messages prometheus check for mirror maker

https://gerrit.wikimedia.org/r/422335

gerritbot added a comment.Mar 27 2018, 8:18 PM

Change 422335 merged by Ottomata:
[operations/puppet@production] Use scalar in dropped messages prometheus check for mirror maker

https://gerrit.wikimedia.org/r/422335

Ottomata claimed this task.Mar 27 2018, 8:18 PM
Ottomata changed the point value for this task from 5 to 8.
Ottomata added a comment.Mar 27 2018, 8:31 PM

lag alert:

MirrorMaker throughput and dropped messages alerts:

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Mar 28 2018, 1:34 PM
gerritbot added a comment.Mar 28 2018, 3:38 PM

Change 422424 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Alert on lag in last 30 minutes, alert mirror maker lag for analytics

https://gerrit.wikimedia.org/r/422424

gerritbot added a comment.Mar 28 2018, 3:39 PM

Change 422424 merged by Ottomata:
[operations/puppet@production] Alert on lag in last 30 minutes, alert mirror maker lag for analytics

https://gerrit.wikimedia.org/r/422424

gerritbot added a comment.Mar 28 2018, 4:26 PM

Change 422430 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Be more lenient about MirrorMaker numDroppedMessages alert

https://gerrit.wikimedia.org/r/422430

gerritbot added a comment.Mar 28 2018, 4:27 PM

Change 422430 merged by Ottomata:
[operations/puppet@production] Be more lenient about MirrorMaker numDroppedMessages alert

https://gerrit.wikimedia.org/r/422430

gerritbot added a comment.Mar 28 2018, 7:04 PM

Change 422467 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] check_kafka_consumer_log - STOP != alert, just bursty topics

https://gerrit.wikimedia.org/r/422467

gerritbot added a comment.Mar 28 2018, 7:10 PM

Change 422467 merged by Ottomata:
[operations/puppet@production] check_kafka_consumer_log - STOP != alert, just bursty topics

https://gerrit.wikimedia.org/r/422467

gerritbot added a comment.Mar 29 2018, 2:30 PM

Change 422939 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use promethues based alert rather than burrow lag check alert

https://gerrit.wikimedia.org/r/422939

gerritbot added a comment.Mar 29 2018, 2:37 PM

Change 422939 merged by Ottomata:
[operations/puppet@production] Use promethues based alert rather than burrow lag check alert

https://gerrit.wikimedia.org/r/422939

gerritbot added a comment.Mar 29 2018, 2:55 PM

Change 422945 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix consuer max lag check query

https://gerrit.wikimedia.org/r/422945

gerritbot added a comment.Mar 29 2018, 2:55 PM

Change 422945 abandoned by Ottomata:
Fix consuer max lag check query

https://gerrit.wikimedia.org/r/422945

Nuria moved this task from Done to In Code Review on the Analytics-Kanban board.Mar 29 2018, 4:04 PM
Ottomata moved this task from In Code Review to Done on the Analytics-Kanban board.Apr 5 2018, 3:41 PM
Nuria closed this task as Resolved.Apr 12 2018, 10:08 PM
Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL