wmcs.openstack.cloudvirt.safe_reboot creates a silence in alertmanager and removes it after rebooting the host.
However, the NeutronAgentDown continues to fire for 20 minutes after the reboot because it uses
min_over_time(openstack_neutron_agent_state{adminState="up"}[20m]) == 0min_over_time[20m] was introduced in this patch to avoid alert flapping.
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| team-wmcs: neutron: Stop using min_over_time | Majavah | operations/alerts | master | +2 -2 |
T335943 decreased the Prometheus scrape interval to be short enough that we can drop the min_over_time() term there. I'm not sure if that's enough to stop the flapping from the cookbook (if it doesn't actually wait for the alert to clear) but should help at least.
Change #1172018 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/alerts@master] team-wmcs: neutron: Stop using min_over_time
Change #1172018 merged by jenkins-bot:
[operations/alerts@master] team-wmcs: neutron: Stop using min_over_time