| elukey | |
| Aug 17 2017, 10:05 AM |
| F23213335: Screen Shot 2018-07-02 at 9.21.39 AM.png | |
| Jul 2 2018, 7:29 AM |
| F23213333: Screen Shot 2018-07-02 at 9.23.10 AM.png | |
| Jul 2 2018, 7:29 AM |
| F23213334: Screen Shot 2018-07-02 at 9.21.56 AM.png | |
| Jul 2 2018, 7:29 AM |
In T172681 no varnishkafka alarm fired and we didn't get any notice of the error until somebody looked in grafana by chance.
I tried in T172681 to tune the alarms but I failed, so I rolled-back to the previous version with a lower critical threshold (5k instead of 20k).
This task should check the following:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| profile::cache::kafka::alerts: move per host vk alarms to aggregates | operations/puppet | production | +93 -57 |
Change 443086 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] icing::monitor::analytics: move per host vk alarms to aggregates
Tried to review the delivery report per second error rate over the past year, to see the use cases that we'd need to take into consideration to establish a good threshold:
The first group of errors on the left corner is T172681, a use case that I'd definitely want to catch. Zooming in:
In this case it seems that a critical threshold of 5 over the course of 20/30 mins should be enough. The other use cases should be caught as well since they have been definitely louder than this one.
Change 443086 merged by Elukey:
[operations/puppet@production] profile::cache::kafka::alerts: move per host vk alarms to aggregates