Prometheus alerting support on Toolforge
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• taavi
	Jun 12 2021, 2:57 PM

Description

Toolforge could use some better tooling for monitoring and alerting than Toolschecker currently provides. Based on a very quick look here are a few requests for better alerting that could be solved with a simple config change if we had a way to send out alerts directly from tools-prometheus:

T215155: Toolforge: systemd monitoring node_systemd_unit_state{state="failed"} == 1
T280741: Add a toolschecker to test the tools email relay
T282738: Alert on CrashLoopBackOff in Toolforge infrastructure pods

I imagine setting a prometheus alertmanager isn't a challenge, but sending out pages and notifications can be somewhat tricky (authentication for acking, victorops keys living on the cloud realm, etc).

Details

	Subject	Repo	Branch	Lines +/-
	P:toolforge::prometheus: deploy alert rules from GitLab	operations/puppet	production	+8 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T321808 Port most/all Icinga checks to Prometheus/Alertmanager
Open	None	T333638 Desired Icinga state by end of FY2023/2024
In Progress	None	T328502 Move WMCS off of Icinga and introduce alertmanager
Open	None	T345983 Remove Icinga checks for Cloud VPS projects (not: infrastructure)
Open	None	T313030 [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones
Resolved	• taavi	T284860 Prometheus alerting support on Toolforge
Resolved	• taavi	T304716 Cloud services enhancement proposal: Prometheus metrics for Toolforge/Toolsbeta/Paws Kubernetes clusters

Event Timeline

• taavi created this task.Jun 12 2021, 2:57 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 12 2021, 2:57 PM

• taavi updated the task description. (Show Details)Jun 12 2021, 3:39 PM

• taavi updated the task description. (Show Details)Jun 12 2021, 3:41 PM

We currently have a separate project of metricsinfra that is running a prometheus instance with alertmanager running. It was thrown together as a proof of concept in T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken and then work on it has not continued. You can view the alerts from that at https://vpsalertmanager.toolforge.org

The problems:

It's not flexible, any monitored projects are normally monitored exactly the same way.
It emails cloud-admin-feed (I think) so basically, WMCS gets emails alerts and that's it. It should email the project admins.
It maintains only short-term metrics.

Some work to improve that is in T266050: Build Prometheus service for use by all Cloud VPS projects and their instances. Toolforge has special needs in all this, but ideally this work should sync up with that work/plan. If you would like to tackle some of it, perhaps you need admin on metricsinfra. Just solving some parts of that ticket with additional config flexibility in puppet would probably resolve this ticket.

The alerts go to https://lists.wikimedia.org/postorius/lists/cloud-admin-feed.lists.wikimedia.org/ if you want to try to subscribe to what is there. It's very generic and does not currently address crash looping pods, email or systemd

• taavi mentioned this in T284938: metricsinfra project admin for Majavah.Jun 14 2021, 4:59 PM

In T284860#7155808, @Bstorm wrote:

The alerts go to https://lists.wikimedia.org/postorius/lists/cloud-admin-feed.lists.wikimedia.org/ if you want to try to subscribe to what is there. It's very generic and does not currently address crash looping pods, email or systemd

I think they never ended up on the list because it had a ban set for ^(?!.*(wikimedia|wikipedia)\.org$) and the mails were apparently coming from root at wmflabs dot org. I changed the regex to ^(?!.*(wikimedia|wikipedia|wmflabs|wmcloud)\.org$).

I used the bulk subscribe feature to add root@wmflabs.org as a user who doesn't get emails but can post to the list without moderation so that we can get alerts with it.

• nskaggs triaged this task as High priority.Aug 10 2021, 10:11 PM

• Bstorm mentioned this in T289364: Review paging setup for WMCS with onboarding in alertmanager.Aug 20 2021, 4:57 PM

• taavi added a subtask: T304716: Cloud services enhancement proposal: Prometheus metrics for Toolforge/Toolsbeta/Paws Kubernetes clusters.Apr 16 2022, 6:50 PM

• taavi added a parent task: T313030: [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones.Jul 14 2022, 2:06 PM

fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:51 PM

fnegri moved this task from Kanban to Inbox on the cloud-services-team board.

Change 890490 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::prometheus: deploy alert rules from GitLab

https://gerrit.wikimedia.org/r/890490

gerritbot added a project: Patch-For-Review.Feb 20 2023, 4:57 PM

• taavi claimed this task.Feb 20 2023, 4:57 PM

• taavi closed this task as Resolved.Feb 28 2023, 10:59 AM