Over the holidays, we had two instances where the "Delay injecting Recent Changes" metric was above the alerting threshold for hours:
What happened?:
The alert emails kept coming and switching between firing and resolving the alert
What should have happened instead?:
One email at the beginning about the alert firing and one email at the end about the alert being resolved.
Other information (browser name/version, screenshots, etc.):
- This used to work properly in the past.
- There is a (maybe unrelated?) task about another Change Dispatching alert not working properly: T349178: [C-DIS] [TECH] "DispatchChanges Normal job backlog time (p50, 15min)" alert misfires with "No Data"