Page MenuHomePhabricator
Create Task
Maniphest T228145

Alert on kafka logging consumer lag
Closed, ResolvedPublic
Actions

Description

As per https://wikitech.wikimedia.org/wiki/Incident_documentation/20190715-logstash we're missing alerts on ongoing kafka queue building up.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
prometheus: add kafka logging consumer lag alertsoperations/puppetproduction+24 -0
Customize query in gerrit

Event Timeline

fgiunchedi created this task.Jul 16 2019, 9:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 16 2019, 9:09 AM
jijiki subscribed.Jul 16 2019, 9:10 AM
gerritbot added a comment.Jul 16 2019, 9:20 AM

Change 523667 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add kafka logging consumer lag alerts

https://gerrit.wikimedia.org/r/523667

gerritbot added a project: Patch-For-Review.Jul 16 2019, 9:20 AM
gerritbot added a comment.Jul 17 2019, 8:09 AM

Change 523667 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add kafka logging consumer lag alerts

https://gerrit.wikimedia.org/r/523667

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2019, 8:10 AM
fgiunchedi closed this task as Resolved.Jul 22 2019, 2:20 PM
fgiunchedi claimed this task.

Completed, we're alerting on consumer lag for kafka logging

fgiunchedi added a project: observability.Aug 19 2019, 2:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits