Route systemd unit alerts to the correct team
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jbond
	Oct 18 2023, 9:33 AM

Description

Currently alerts for failed systemd units are routed to specific teams based on the role owner, in many cases this is correct however there are some instances where this is not the correct thing to do. one example of this are the various httpbb_* checks that run on the cumin host. Theses checks belong to the service ops team but they run on a machine owned by the Infrastructure foundations team.

On solution to theses specific alerts would be to move the checks to a machine owned by service ops which would work if this is the only such alert. however i suspect there are likely others as such it would be nice to somehow send a signal from the systemd unit to override the owner.

Details

Subject	Repo	Branch	Lines +/-
team-sre: ignore systemd_unit_.+_owner stale textfile	operations/alerts	master	+1 -1
team-sre/systemd: update systemd checks to make use of systemd_unit_owner	operations/alerts	master	+25 -3
docker::reporter: route k8s alerts to service ops	operations/puppet	production	+4 -5
systemd::service: Add service owner parameter	operations/puppet	production	+39 -5

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T321808 Port most/all Icinga checks to Prometheus/Alertmanager
Resolved	fgiunchedi	T332709 Multi-team Prometheus/Alertmanager alerts
Resolved	jbond	T349176 Route systemd unit alerts to the correct team

Event Timeline

jbond created this task.Oct 18 2023, 9:33 AM

Thanks for the task! I think another potential use case are the docker-reporter* units on the build host.

Indeed thank you @jbond for the task. The "signal" we could send is to be able to associate a team to a unit at the puppet level, then at the host level gather such information and export it into a per-unit metric, akin to role_owner (e.g. systemd_unit_owner). The logic then at the alert level would be similar as role_owner, plus catering for the fact that a missing systemd_unit_owner should fallback to role_owner.

Change 968293 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] systemd::service: Add service owner parameter

https://gerrit.wikimedia.org/r/968293

gerritbot added a project: Patch-For-Review.Oct 24 2023, 4:21 PM

Change 968293 merged by Jbond:

[operations/puppet@production] systemd::service: Add service owner parameter

https://gerrit.wikimedia.org/r/968293

Maintenance_bot removed a project: Patch-For-Review.Oct 26 2023, 1:30 PM

Change 969121 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] docker::reporter: route k8s alerts to service ops

https://gerrit.wikimedia.org/r/969121

gerritbot added a project: Patch-For-Review.Oct 26 2023, 1:45 PM

Change 969121 merged by Jbond:

[operations/puppet@production] docker::reporter: route k8s alerts to service ops

https://gerrit.wikimedia.org/r/969121

Maintenance_bot removed a project: Patch-For-Review.Oct 26 2023, 4:31 PM

Change 969312 had a related patch set uploaded (by Jbond; author: jbond):

[operations/alerts@master] team-sre/systemd: update systemd checks to make use of systemd_unit_owner

https://gerrit.wikimedia.org/r/969312

gerritbot added a project: Patch-For-Review.Oct 27 2023, 10:35 AM

Change 969312 merged by Jbond:

[operations/alerts@master] team-sre/systemd: update systemd checks to make use of systemd_unit_owner

https://gerrit.wikimedia.org/r/969312

Maintenance_bot removed a project: Patch-For-Review.Oct 27 2023, 12:30 PM

We now have a team parameter on the systemd::unit, systemd::service and systemd::timer::job which should allow us to correctly route alerts. this has already been configured for the units mentioned in this task

Change 970402 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] team-sre: ignore systemd_unit_.+_owner stale textfile

https://gerrit.wikimedia.org/r/970402

gerritbot added a project: Patch-For-Review.Oct 31 2023, 3:53 PM

Change 970402 merged by Filippo Giunchedi:

[operations/alerts@master] team-sre: ignore systemd_unit_.+_owner stale textfile

https://gerrit.wikimedia.org/r/970402

Maintenance_bot removed a project: Patch-For-Review.Nov 1 2023, 9:10 AM

lmata moved this task from Inbox to Done on the SRE Observability (FY2023/2024-Q2) board.Jan 26 2024, 1:08 AM