Page MenuHomePhabricator

Add 'source' tag to icinga and prometheus/thanos alerts
Closed, ResolvedPublic

Description

As per T293209 we want to add source label to alerts, specifically:

Event Timeline

Change 763459 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/debs/prometheus-icinga-exporter@master] am: remove Icinga/ prefix and add 'source' label

https://gerrit.wikimedia.org/r/763459

Change 763460 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/debs/prometheus-icinga-exporter@master] am: add 'host' label and add port to 'instance'

https://gerrit.wikimedia.org/r/763460

Change 763697 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] thanos: add relabels to rule

https://gerrit.wikimedia.org/r/763697

Change 763698 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: inject 'source' label to alerts

https://gerrit.wikimedia.org/r/763698

Change 764304 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] alertmanager: route source=icinga alerts

https://gerrit.wikimedia.org/r/764304

Change 763459 merged by Filippo Giunchedi:

[operations/debs/prometheus-icinga-exporter@master] am: remove Icinga/ prefix and add 'source' label

https://gerrit.wikimedia.org/r/763459

Change 763697 merged by Filippo Giunchedi:

[operations/puppet@production] thanos: add relabels to rule

https://gerrit.wikimedia.org/r/763697

Change 763698 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: inject 'source' label to alerts

https://gerrit.wikimedia.org/r/763698

Change 764304 merged by Filippo Giunchedi:

[operations/puppet@production] alertmanager: route source=icinga alerts

https://gerrit.wikimedia.org/r/764304

Mentioned in SAL (#wikimedia-operations) [2022-02-21T09:24:01Z] <godog> deploy prometheus-icinga-exporter 0.19 - T300951

fgiunchedi claimed this task.

This is complete! Now icinga/prometheus/thanos alerts have their source label attached.

An exception for now are grafana alerts which AFAIK doesn't support adding a label for all outgoing alerts; I'll punt on the decision on what to do for grafana alerts for now

Change 763460 abandoned by Filippo Giunchedi:

[operations/debs/prometheus-icinga-exporter@master] am: add 'host' label and add port to 'instance'

Reason:

Abandoning this for now, not needed

https://gerrit.wikimedia.org/r/763460