Page MenuHomePhabricator

Refine the systemd alerts to focus on critical services
Closed, ResolvedPublic

Description

Write the description below

We get many pages due to systemd failed units, that end up being non-critical (ex. debmonitor) and can wait for the
next person awake to get resolved.

Currently the check can't be made more granular, but it can be disabled:

See https://gerrit.wikimedia.org/g/operations/puppet/+/refs/heads/production/modules/profile/manifests/base.pp#14

Boolean $monitor_systemd = lookup('profile::base::monitor_systemd', {default_value => true}),

But that will require to make sure we are still explicitly checking the critical service.

This task is to:

  • allow some discussion
  • gather the list of those services
  • add an explicit check for each of them
  • remove the generic check

This can be done in batches (ex. first cloudvirts, then cloudcontrols, then cephosds...) so we start avoiding extra
pages sooner than later.

Event Timeline

dcaro triaged this task as High priority.

Change 713909 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] openstack: stop paging for systemd alone

https://gerrit.wikimedia.org/r/713909

Change 713909 merged by Bstorm:

[operations/puppet@production] openstack: stop paging for systemd alone

https://gerrit.wikimedia.org/r/713909

Change 714087 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] openstack: correct the hiera entry for the systemd alert disablement

https://gerrit.wikimedia.org/r/714087

Change 714087 merged by Bstorm:

[operations/puppet@production] openstack: correct the hiera entry for the systemd alert disablement

https://gerrit.wikimedia.org/r/714087

dcaro claimed this task.

It's not relevant anymore with the move to prometheus, as the generic alerts are warning only.