As per T293209 we want to add source label to alerts, specifically:
- For Icinga; strip Icinga/ prefix and add source in icinga-exporter
- For Prometheus use alert relabel config
- Ditto for Thanos rule via relabel config file
As per T293209 we want to add source label to alerts, specifically:
Change 763459 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/debs/prometheus-icinga-exporter@master] am: remove Icinga/ prefix and add 'source' label
Change 763460 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/debs/prometheus-icinga-exporter@master] am: add 'host' label and add port to 'instance'
Change 763697 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] thanos: add relabels to rule
Change 763698 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] prometheus: inject 'source' label to alerts
Change 764304 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] alertmanager: route source=icinga alerts
Change 763459 merged by Filippo Giunchedi:
[operations/debs/prometheus-icinga-exporter@master] am: remove Icinga/ prefix and add 'source' label
Change 763697 merged by Filippo Giunchedi:
[operations/puppet@production] thanos: add relabels to rule
Change 763698 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: inject 'source' label to alerts
Change 764304 merged by Filippo Giunchedi:
[operations/puppet@production] alertmanager: route source=icinga alerts
Mentioned in SAL (#wikimedia-operations) [2022-02-21T09:24:01Z] <godog> deploy prometheus-icinga-exporter 0.19 - T300951
This is complete! Now icinga/prometheus/thanos alerts have their source label attached.
An exception for now are grafana alerts which AFAIK doesn't support adding a label for all outgoing alerts; I'll punt on the decision on what to do for grafana alerts for now
Change 763460 abandoned by Filippo Giunchedi:
[operations/debs/prometheus-icinga-exporter@master] am: add 'host' label and add port to 'instance'
Reason:
Abandoning this for now, not needed