Page MenuHomePhabricator

Try out new Grafana alert setup
Open, Needs TriagePublic

Description

In Grafana 8.0 the alert functionality was overhauled and in 8.3 the new alert system is the default. Let me try that out and add documentation on how it works and how we can configure it for performance alerts when Grafana is upgraded.

Related Objects

Event Timeline

I've been trying out the new alerts in Grafana 8.3.3. I tried setting up a couple of alerts to get a feeling for how it works and what will be the difference when we move over to the new version.

The new alerts are not tied to a specific graph, instead you store the alerts in a folder of your choice with only the data needed for the alert. The GUI looks like this:

Screenshot 2022-01-04 at 12.46.47.png (1×2 px, 672 KB)

I've used the "classic condition" as we use today to setup alerts. That works fine to mimic the current behaviour. However the new alerts makes it easier/more logic to get data from difference datasources and make alerts that fires on multiple conditions. I need to try that too to see what kind of setup we can have:

Screenshot 2022-01-04 at 12.47.22.png (1×2 px, 180 KB)

When an alert is fired, you can have your own template of how the alert is generated (using Go templates). That is nice, when I tried it with Slack alerts, I had some problem reverting back to old behaviours and it some documentation missing but my guess is that it will come in later releases.

One major thing that is difference than how we do things today, is that the alert is not viewable in the graphs in any way. An alert annotation do not have any tags, so you cannot create a annotation query to show it in any graphs. It was the same in old alerts (you could not show alerts in any dashboard that you want), only show the alert in the specific dashboard where it was created. But since now the alert do not belong to any dashboard, so you cannot see it. That makes it a little bit harder to debug alerts, since you cannot see exactly when it fired, but I hope that functionality will come soon.