We have detected a number of cases in which prometheus metrics seems to disappear for no apparent reason, thus causing bogus alerts.
Some examples:
- the CephClusterInUnknown alert uses this:
expr: (ceph_health_status{job="ceph_eqiad"} or on() vector(-1)) == -1
In grafana we can see this:
Base metric:
Alert: (I interpret 'no data' as: there should be no alert)
However, the alert is firing in alertmanager:
- the OpenstackAPIResponse alert uses this:
expr: avg_over_time((haproxy_server_total_time_average_seconds{job="cloudlb-haproxy", proxy!~"(mysql|wikireplica-db-(web|analytics)-s\\d)"} OR on() vector(100))[12h:]) > 99
In grafana we see this:
Base metric:
Alert expression: (again, I interpret 'no data' as there should be no alert triggering)
However, the alert is showing in alertmanager:
- The Neutron alerts reported in T374513: Lint problems for NeutronAgentDownForLong and NeutronAgentDown and T373878: openstack: fix missing prometheus metrics
Apparently, same behavior as with the other two.
