Icinga downtimes not working
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	MoritzMuehlenhoff
	Aug 2 2022, 6:06 AM

Description

During maintenance on db1181 "profile::monitoring::notifications_enabled: false" was set, but it still paged. Similar report by @RKemper on IRC from last night:

<ryankemper> Anyone been having trouble downtiming hosts in Icinga? For example I just tried manually downtiming `elastic2059` and never saw it actually take effect in Icinga
<ryankemper> Feels like Icinga is getting the request fine but then failing to actually process it in the backend
<ryankemper> ^ Following up I now see those downtimes in place so it does seem like the icinga backend is just really backlogged

Followups

(critical) Alert on "max check latency" shooting up
Depending on how frequently the above fires, consider auto-remediation (i.e. an Icinga restart)

Details

	Subject	Repo	Branch	Lines +/-
	o11y: alert on Icinga max check latency	operations/alerts	master	+26 -0

Customize query in gerrit

Related Objects

Mentioned In: T196336: Icinga passive checks go awol and downtime stops working
Mentioned Here: T196336: Icinga passive checks go awol and downtime stops working

Event Timeline

MoritzMuehlenhoff created this task.Aug 2 2022, 6:06 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 2 2022, 6:06 AM

Happens since yesterday around 16:30 UTC: https://grafana.wikimedia.org/d/rsCfQfuZz/icinga?orgId=1&from=now-24h&to=now

Mentioned in SAL (#wikimedia-operations) [2022-08-02T06:46:20Z] <godog> bounce icinga on alert1001 - T314353

Indeed check max latency spiked up to 30+ min (!) around that time

Is this just a duplicate/continuation of T196336 ?

Mentioned in SAL (#wikimedia-operations) [2022-08-02T07:22:36Z] <godog> bounce icinga on alert2001 - T314353

In T314353#8122434, @jcrespo wrote:

Is this just a duplicate/continuation of T196336 ?

It is definitely possible, I don't know ATM

A restart of Icinga on alert1001 brought things back AFAICT, the "max check latency" (which is so far the one/only signal I found of sth being wrong) went back down within ~30 min of Icinga restarting and it is now around 60s as usual.

fgiunchedi updated the task description. (Show Details)Aug 2 2022, 7:31 AM

fgiunchedi added a project: SRE Observability (FY2022/2023-Q1).

Peachey88 updated the task description. (Show Details)Aug 2 2022, 8:00 AM

lmata subscribed.Aug 2 2022, 12:15 PM

In T314353#8122412, @fgiunchedi wrote:

Indeed check max latency spiked up to 30+ min (!) around that time

I feel like T196336 is/was also caused by general latency.

Dzahn mentioned this in T196336: Icinga passive checks go awol and downtime stops working.Aug 2 2022, 7:22 PM

Change 820072 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] o11y: alert on Icinga max check latency

https://gerrit.wikimedia.org/r/820072

gerritbot added a project: Patch-For-Review.Aug 3 2022, 7:50 AM

fgiunchedi added a project: User-fgiunchedi.Aug 3 2022, 12:14 PM

lmata moved this task from Inbox to In progress on the SRE Observability (FY2022/2023-Q1) board.Aug 3 2022, 2:55 PM

Change 820072 merged by Filippo Giunchedi:

[operations/alerts@master] o11y: alert on Icinga max check latency

https://gerrit.wikimedia.org/r/820072

We are now alerting on elevated max check latency, I'm going to stall the task and re-evaluate in a couple of months if we need to deploy auto-remediation as well.

fgiunchedi moved this task from Backlog to Radar on the User-fgiunchedi board.Aug 4 2022, 8:04 AM

lmata triaged this task as Medium priority.Sep 6 2022, 6:00 PM

herron subscribed.Sep 6 2022, 6:07 PM

lmata moved this task from Inbox to Prioritized on the Observability-Alerting board.Sep 6 2022, 6:46 PM

max check alert latency did shoot up over last weekend (though to 7min) and self-recovered as far as I can tell:

2022-10-10-100445_1176x327_scrot.png (327×1 px, 39 KB)

except for that spike it looks like check latency is under control (and going down, as we progressively remove more and more check from icinga). I'm optimistically resolving the task

lmata moved this task from Prioritized to Done on the Observability-Alerting board.Jan 16 2023, 5:57 PM

	F35560029: 2022-10-10-100445_1176x327_scrot.png
	Oct 10 2022, 8:05 AM

Icinga downtimes not workingClosed, ResolvedPublicActions