Page MenuHomePhabricator

Karma UI shows duplicate alerts
Open, Needs TriagePublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

Open the Karma UI at https://alerts.wikimedia.org/

What happens?:

Some alerts are shown twice, once with receiver=default and once with receiver={customreceiver}

This only seems to happen for alerts with severity=warning.

Example:

Screenshot 2023-12-14 at 16.03.32.png (866×2 px, 201 KB)

This alert has severity=critical and a custom receiver (though it is not shown in the screenshot for some reason) and is only displayed once:

Screenshot 2023-12-14 at 16.07.34.png (702×1 px, 95 KB)

Other information

There is a related issue upstream (https://github.com/prymitive/karma/issues/5144) but I think that describes a different problem, when an alert is displayed twice with two custom receivers, not one custom receiver + the default one.

An example would be this one:

Screenshot 2023-12-14 at 16.12.27.png (838×2 px, 163 KB)

Event Timeline

Thank you for looking into this @fnegri !

I've looked and the alertmanager configuration and the upstream issue again and I think what's going on is:

  • The duplication happens only when there's > 1 receiver for a given alert
  • The default receiver is both the default for the route entry (i.e. the root of the routing tree) and also as a "catch all" as the last route of routes (i.e. the first level of tree branches)
  • For certain kind of alerts we continue in the routing tree, this can cause an alert to have multiple receivers
  • There's nothing special about default receiver, i.e. it is just named default but could be named anything

Given the above the behavior is consistent with our configuration and the upstream issue, e.g. icinga alerts are effectively blackholed immediately and thus never get the default receiver.

Something else to keep in mind is that unique labels across all alerts being displayed (that are also not in the search bar) will be hidden, this is why for example @receiver is not displayed here (all icinga alerts) https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=source%3Dicinga
But it is displayed when looking at prometheus alerts https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=source%3Dprometheus

HTH!

I'll leave the task open to track this issue in karma (duplicated receiver)