Page MenuHomePhabricator
Create Task
Maniphest T285205

Make sure LibreNMS -> AM alerts don't flap due to lack of notifications
Closed, ResolvedPublic
Actions

Description

We've been running recently into an issue where a few console servers have CPU at 100%, the related librenms alerts (and thus AM notifications) were flapping at regular intervals, e.g.

05:07 -jinxer-wm:#wikimedia-operations- (Processor usage over 85%) firing: Processor usage over 85%   - 
          https://alerts.wikimedia.org
05:12 -jinxer-wm:#wikimedia-operations- (Processor usage over 85%) resolved: Processor usage over 85%   - 
          https://alerts.wikimedia.org
07:37 -jinxer-wm:#wikimedia-operations- (Processor usage over 85%) firing: Processor usage over 85%   - 
          https://alerts.wikimedia.org
07:42 -jinxer-wm:#wikimedia-operations- (Processor usage over 85%) resolved: Processor usage over 85%   - 
          https://alerts.wikimedia.org
07:47 -jinxer-wm:#wikimedia-operations- (Processor usage over 85%) firing: Processor usage over 85%   - 
          https://alerts.wikimedia.org
07:57 -jinxer-wm:#wikimedia-operations- (Processor usage over 85%) firing: Processor usage over 85%   - 
          https://alerts.wikimedia.org
08:02 -jinxer-wm:#wikimedia-operations- (Processor usage over 85%) resolved: Processor usage over 85%   - 
          https://alerts.wikimedia.org

This is due to the interaction between librenms' interval for re-sending the alerts (e.g. 3h for the alert above) and the fact that AM expects clients to keep sending notifications while the alerts are active.

AFAICT AM's expectation is maximum ~5 minutes between clients sending notifications, while librenms poller interval we're using ATM is also 5 minutes.

Details

SubjectRepoBranchLines +/-
alertmanager: bump resolve_timeout to 20moperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Jun 21 2021, 8:09 AM
fgiunchedi added a comment.Jun 21 2021, 8:54 AM

The solution I think will work:

  • Set alertmanager resolve_timeout (docs) to say 20m. This affects only alerts with no endsAt field, which librenms doesn't set
  • Make sure interval for librenms alerts is set to e.g. 15m
gerritbot added a comment.Jun 21 2021, 8:59 AM

Change 700482 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] alertmanager: bump resolve_timeout to 20m

https://gerrit.wikimedia.org/r/700482

gerritbot added a project: Patch-For-Review.Jun 21 2021, 8:59 AM
gerritbot added a comment.Jun 21 2021, 9:05 AM

Change 700482 merged by Filippo Giunchedi:

[operations/puppet@production] alertmanager: bump resolve_timeout to 20m

https://gerrit.wikimedia.org/r/700482

Maintenance_bot removed a project: Patch-For-Review.Jun 21 2021, 9:10 AM
Stashbot added a comment.Jun 21 2021, 12:55 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-21T12:55:21Z] <godog> move librenms alerts with "max alerts" == -1 to "interval" being 15m - T285205

fgiunchedi added a subscriber: cmooney.Jun 21 2021, 1:01 PM

I've adjusted all LibreNMS "persisting" alerts (i.e. those with max alerts == -1 and thus librenms will keep sending notifications) to have "interval" set to 15m, which does avoid the flapping alerts. In practice LibreNMS will send notifications to AM for these alerts faster than AM will consider them "stale" (and thus resolved, after 20m).

There are a bunch of alerts with "max alerts" == 1 (i.e. that will notify only once), we'll need to go through those with @ayounsi and @cmooney and figure out what the expectations are for those.

lmata subscribed.Jun 21 2021, 1:50 PM
lmata moved this task from Inbox to In progress on the observability board.Jun 21 2021, 3:36 PM
ayounsi added a comment.Jun 21 2021, 4:54 PM

There are a bunch of alerts with "max alerts" == 1 (i.e. that will notify only once), we'll need to go through those with @ayounsi and @cmooney and figure out what the expectations are for those.

Changed them both from 0 to 900.

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jun 22 2021, 12:47 PM
lmata added a project: SRE Observability (FY2021/2022-Q1).Jul 12 2021, 2:19 AM
lmata removed a project: observability.
lmata moved this task from Inbox to In progress on the SRE Observability (FY2021/2022-Q1) board.
fgiunchedi closed this task as Resolved.Jul 20 2021, 10:01 AM
fgiunchedi claimed this task.

This is done!

lmata moved this task from In progress to Done on the SRE Observability (FY2021/2022-Q1) board.Sep 14 2021, 9:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits