Increased icinga check latency since 05/12
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Dec 7 2020, 9:26 AM

Description

Looks like icinga check latency increased (max/avg) increased a few days ago and didn't go back down:

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 7 2020, 9:26 AM

lmata moved this task from Inbox to In progress on the observability board.

Mentioned in SAL (#wikimedia-operations) [2020-12-07T18:52:54Z] <herron> systemctl restart icinga on alert1001 T269560

Starting around 1400 UTC today, average check latency has been dropping steadily.

Puppet changes prior to and around that time do not correlate with symptoms (increasing check attempts on maps cluster after 1400)
Host overview of alert1001 shows a drop in system load and network utilization for the duration of the event which appears to end around the restart at 1852 UTC today

Checking back in some time later, this does not appear to have occurred again.

	F33938506: 2020-12-07-102632_1189x350_scrot.png
	Dec 7 2020, 9:26 AM