I have added a new Grafana panel showing the state of that specific check on https://grafana.wikimedia.org/dashboard/db/production-logging Production Logging.

Direct link is https://grafana.wikimedia.org/dashboard/db/production-logging?panelId=19&fullscreen

Ori is out for family reasons right now, but since he helped craft this alert I'm adding him here for his thoughts.

The alarm is not delayed by half an hour. It is delay but not as much as I thought.

Looking at the green/orange/red graph, one can see the huge spike at 12:15 which correlate with Icinga:

There are two issues with this one:

1. it is 7 minutes late
2. it is in SOFT state indicating that Icinga is set to retry this X (actually 3) times before becoming HARD and triggering notification

The state got cleared exactly 2 minutes later when the spike ended.

The 1st IRC notification we received (at 12:50:18) is unrelated to the spike at 12:15. It is a second spike that occured at 12:40. It prompted me to look at the log and I thought it was for the previous spike hence complaining about more than half an hour of delay.

From Icinga we can see there are 3 retries each 2 minutes apparts before emitting the notification:

The things to look at are:

• reduce the amount and or delay of retries
• figure out why even the first step change is lagged out.

From the three spikes, one got unnotified and the two others delayed by 10 minutes.

Indeed, there is a bit of delay due to retries and the default retry_interval of 1 (minute) which seems appropriate for most cases.

@hashar does this still need looked at/addressed? If so, how much delay do you think is acceptable or appropriate?

In a nutshell the first issue is that the first spike of error at 12:15 was large in number of events but happened on a very short window. The statsd check was made on a larger window and less than 50 % of the points were failing so it did not trigger an error.

Assuming a spike that spans 3 units of time, and a statsd check that spans 7 units:

errors
|          x
|          x
|         xxx
|         xxx
+---->0000111<

There are 3 points flagged as error (1 above), 4 which are fine (0 above) that less than 50% of data points in error state and the spike is left unnoticed.

The second issue is similar, the spike is noticed a bit late since it has to occupy the statsd check window until 50% of data points are erroring out but on top of that there are the Icinga retries that further delay the SOFT > HARD state which will eventually trigger the notification. There was 3 checks 2 minutes apart so the notification is send 6 minutes after the statsd window is marked in error.

I guess this task was to have Icinga to issue the notification immediately (0 retries) since the statsd window acts a buffer already. Then all of that was more than 4 years ago so should probably be entirely revisited and indeed we can just ignore the issue unless that becomes a concern again.