⚓ T313030 [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones

Subject	Repo	Branch	Lines +/-
P:toolforge::checker: remove webservice checks	operations/puppet	production	+2 -84
P:toolforge::checker: remove kubernetes node readiness check	operations/puppet	production	+7 -70
P:toolforge::checker: remove ToolsDB R/W check	operations/puppet	production	+12 -67

	Title	Reference	Author	Source Branch	Dest Branch
	kubernetes: Add node ready alerts	repos/cloud/toolforge/alerts!5	taavi	node-ready	main

Status	Assigned	Task
Open	None	T321808 Port most/all Icinga checks to Prometheus/Alertmanager
Open	None	T333638 Desired Icinga state by end of FY2023/2024
In Progress	taavi	T328502 Move WMCS off of Icinga and introduce alertmanager
Open	None	T345983 Remove Icinga checks for Cloud VPS projects (not: infrastructure)
Open	None	T313030 [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones
Resolved	taavi	T311911 Upgrade Toolforge Prometheus servers to Debian Bullseye or later
Resolved	• nskaggs	T312692 Request increased quota for tools Cloud VPS project
Resolved	taavi	T284860 Prometheus alerting support on Toolforge
Resolved	taavi	T304716 Cloud services enhancement proposal: Prometheus metrics for Toolforge/Toolsbeta/Paws Kubernetes clusters
Open	taavi	T288053 Add external meta-monitoring for metricsinfra
Resolved	dcaro	T313031 Paging for alerts from the cloud realm
Open	None	T357977 [toolforge.infra] create fullstack tests
Resolved	taavi	T358333 Remove toolschecker grid engine checks

taavi created this task.Jul 14 2022, 2:06 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 14 2022, 2:06 PM

taavi added subtasks: T311911: Upgrade Toolforge Prometheus servers to Debian Bullseye or later, T284860: Prometheus alerting support on Toolforge.Jul 14 2022, 2:06 PM

taavi added a subtask: T288053: Add external meta-monitoring for metricsinfra.Jul 14 2022, 2:09 PM

taavi closed subtask T311911: Upgrade Toolforge Prometheus servers to Debian Bullseye or later as Resolved.Sep 23 2022, 12:59 PM

fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:38 PM

fnegri moved this task from Kanban to Inbox on the cloud-services-team board.

taavi closed subtask T284860: Prometheus alerting support on Toolforge as Resolved.Feb 28 2023, 10:59 AM

dcaro closed subtask T313031: Paging for alerts from the cloud realm as Resolved.Mar 6 2023, 6:29 PM

dcaro changed the status of subtask T313031: Paging for alerts from the cloud realm from Duplicate to Resolved.Apr 11 2023, 9:46 AM

fnegri mentioned this in T344298: mysqld killed by oomkiller on tools-db-1.tools.eqiad1.wikimedia.cloud.Aug 22 2023, 3:34 PM

taavi added a parent task: T345983: Remove Icinga checks for Cloud VPS projects (not: infrastructure).Sep 9 2023, 7:44 PM

taavi updated the task description. (Show Details)Sep 9 2023, 7:47 PM

Change 956071 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::checker: remove ToolsDB R/W check

https://gerrit.wikimedia.org/r/956071

gerritbot added a project: Patch-For-Review.Sep 9 2023, 7:52 PM

Change 956071 merged by Majavah:

[operations/puppet@production] P:toolforge::checker: remove ToolsDB R/W check

https://gerrit.wikimedia.org/r/956071

Maintenance_bot removed a project: Patch-For-Review.Sep 11 2023, 8:11 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/5

kubernetes: Add node ready alerts

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/5

kubernetes: Add node ready alerts

Maintenance_bot removed a project: Patch-For-Review.Dec 12 2023, 4:30 PM

Change 982786 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::checker: remove kubernetes node readiness check

https://gerrit.wikimedia.org/r/982786

gerritbot added a project: Patch-For-Review.Dec 13 2023, 12:32 PM

Change 982786 merged by Majavah:

[operations/puppet@production] P:toolforge::checker: remove kubernetes node readiness check

https://gerrit.wikimedia.org/r/982786

Maintenance_bot removed a project: Patch-For-Review.Dec 13 2023, 1:10 PM

taavi updated the task description. (Show Details)Jan 16 2024, 1:48 PM

Change 991289 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::checker: remove webservice checks

https://gerrit.wikimedia.org/r/991289

gerritbot added a project: Patch-For-Review.Jan 17 2024, 9:27 AM

Change 991289 merged by Majavah:

[operations/puppet@production] P:toolforge::checker: remove webservice checks

https://gerrit.wikimedia.org/r/991289

Maintenance_bot removed a project: Patch-For-Review.Jan 17 2024, 11:30 AM

dcaro triaged this task as Medium priority.Feb 20 2024, 1:14 PM

dcaro moved this task from Backlog to Workspace for triaging whenever needed on the Toolforge board.

dcaro added a subtask: T357977: [toolforge.infra] create fullstack tests.Feb 20 2024, 1:32 PM

dcaro updated the task description. (Show Details)

fgiunchedi subscribed.Feb 20 2024, 3:03 PM

dcaro renamed this task from Replace Toolschecker alerts with Prometheus based ones to [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones.Feb 21 2024, 10:23 AM

dcaro moved this task from Workspace for triaging whenever needed to Ready to be worked on on the Toolforge board.Feb 21 2024, 4:02 PM

taavi closed subtask T358333: Remove toolschecker grid engine checks as Resolved.Feb 23 2024, 1:42 PM

taavi updated the task description. (Show Details)Feb 23 2024, 1:44 PM

taavi mentioned this in T314664: [infra] Decommission the Grid Engine infrastructure.Mar 13 2024, 12:51 PM