Page MenuHomePhabricator
Create Task
Maniphest T425339

Alert in need of triage: Kafka MirrorMaker main-codfw_to_main-eqiad dropped message count in last 30m (instance alert1002)
Closed, ResolvedPublic
Actions

Description

The alert Kafka MirrorMaker main-codfw_to_main-eqiad dropped message count in last 30m has started firing 1 month ago.

Labels
alertname=Kafka MirrorMaker main-codfw_to_main-eqiad dropped message count in last 30m
instance=alert1002
severity=unknown
source=icinga
team=sre
Annotations
NameContent
dashboardhttps://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-datasource=eqiad+prometheus%2Fops&var-lag_datasource=codfw+prometheus%2Fops&var-mirror_name=main-codfw_to_main-eqiad
runbookhttps://wikitech.wikimedia.org/wiki/Kafka/Administration
summaryNaN
Links
Triage metadata. Do not delete.
fingerprint=3707c33a852895e2

Event Timeline

LSobanski created this task.May 4 2026, 11:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 4 2026, 11:09 AM
LSobanski added a comment.May 4 2026, 11:11 AM

Alerts mention both main and jumbo so tagging both ServiceOps new and Data-Platform-SRE

LSobanski added projects: ServiceOps new, Data-Platform-SRE.May 4 2026, 11:11 AM
JMeybohm added subscribers: elukey, brouberol, JMeybohm.May 4 2026, 3:34 PM

We already tried to remove the icinga alerts completely:
https://gerrit.wikimedia.org/r/c/operations/puppet/+/1282092

But it seems like they are still used, so @elukey reverted it:
https://gerrit.wikimedia.org/r/c/operations/puppet/+/1282335

JMeybohm added a comment.May 6 2026, 7:40 AM

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1282919 fixed a bunch of these alerts, but 3 are still firing

elukey added a comment.May 7 2026, 8:04 AM

All resolved now!

JMeybohm closed this task as Resolved.May 7 2026, 10:05 AM
JMeybohm claimed this task.
Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE.May 8 2026, 8:49 AM
Gehel moved this task from Backlog - project to Done on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.May 8 2026, 8:50 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits