Page MenuHomePhabricator
Create Task
Maniphest T288622

All Prometheus based alerts move from Icinga to alert manager exclusively
Open, MediumPublic
Actions

Description

This is a tracking task to move Prometheus-based (i.e. check_prometheus) alerts from Icinga to Alertmanager. These alerts are the most straightforward to port to Prometheus alerting rules since all information by definition is already available in the form of Prometheus metrics.

Details

Other Assignee
• lmata
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
icinga: remove widespread puppet agent alertsHerronoperations/puppetproduction+0 -25
opensearch: remove icinga::monitor::elasticsearch::old_jvm_gc_checksFilippo Giunchedioperations/puppetproduction+0 -19
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpentappofT395441 Port all Icinga checks to Prometheus/Alertmanager: preparation
 OpentappofT321808 Port all Icinga checks to Prometheus/Alertmanager
 OpentappofT288622 All Prometheus based alerts move from Icinga to alert manager exclusively
 ResolvedfgiunchediT281454 Onboard teams with Prometheus-based alerts to AlertManager
 Resolved• lmataT281359 Onboard teams with Grafana alerts to AlertManager
 ResolvedfgiunchediT282806 Port traffic/netops grafana alerts to AlertManager
 ResolvedfgiunchediT288726 Move > 60% of observability Prometheus-based checks to Alertmanager
 ResolvedNoneT309007 Migrate eventlogging check_prometheus checks to alertmanager
 ResolvedArnoldokothT309009 Migrate eventgate check_prometheus checks to alertmanager
 ResolvedfgiunchediT309010 Migrate Kafka prometheus alerts from Icinga to Alertmanager
 DuplicateNoneT309011 Migrate labstore prometheus alerts from Icinga to Alertmanager
 ResolvedGoaltappofT309012 Migrate zookeeper prometheus checks from Icinga to Alertmanager
 ResolvedNoneT285328 Migrate OSM sync alerts from icinga to AlertManager
 ResolvedEBernhardsonT289077 Migrate Search team's prometheus-based alerts from Icinga to alert-manager
 ResolvedBTullisT293399 Migrate the majority of the analytics cluster alerts from Icinga to AlertManager
 ResolvedjbondT294564 Migrate Foundations Prometheus alerts to AlertManager
 ResolvedjhathawayT302687 Don't move filesystem_avail_bigger_than_size icinga check to alert manager?
 ResolvedBCornwallT300723 Migrate Traffic Prometheus alerts from Icinga to Alertmanager
 ResolvedBCornwallT311445 Create vm.max_map_count metrics for Prometheus
 DuplicateNoneT312763 Port kubernetes prometheus-based alerts from icinga to alertmanager
 ResolvedfgiunchediT312764 Port mediawiki prometheus-based alerts from icinga to alertmanager
 ResolvedfgiunchediT312765 Port swift prometheus-based alerts from icinga to alertmanager
 ResolvedJMeybohmT311251 Migrate kubernetes alerts away from icinga
ResolvedBCornwallT327791 Move (or delete?) trafficserver restart count alert from icinga to alerts.git
 ResolvedfgiunchediT327792 Port k8s cache alerts from icinga to alerts.git
 ResolvedcmooneyT327793 Port (or delete) udpmxircecho alerts from icinga to alerts.git
 ResolvedGoaltappofT370153 Move kafka-mirror Prometheus-based alerts from Icinga to alerts.git
 ResolvedtappofT381665 module to define custom Prometheus alerts directly in Puppet
 ResolvedtappofT425621 Move Kafka mirror monitors and alerts to the alerts repo

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 11 2021, 1:21 PM
• lmata added a project: Goal.Aug 11 2021, 1:21 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2021/2022-Q1) board.Aug 11 2021, 1:24 PM
• lmata updated Other Assignee, added: • lmata.Aug 11 2021, 1:28 PM
fgiunchedi added subtasks: T281454: Onboard teams with Prometheus-based alerts to AlertManager, T281359: Onboard teams with Grafana alerts to AlertManager, T225140: Icinga alerts that should open tasks instead of alerting.Aug 12 2021, 9:00 AM
• lmata awarded a token.Aug 12 2021, 6:32 PM
RKemper removed a subtask: T289077: Migrate Search team's prometheus-based alerts from Icinga to alert-manager.Aug 17 2021, 6:47 PM
fgiunchedi closed subtask T281359: Onboard teams with Grafana alerts to AlertManager as Resolved.Sep 28 2021, 11:48 AM
• lmata triaged this task as Medium priority.Sep 30 2021, 9:51 PM
• lmata edited projects, added: SRE Observability (FY2021/2022-Q2); removed: SRE Observability (FY2021/2022-Q1).
• lmata updated the task description. (Show Details)
• lmata edited projects, added: SRE Observability (FY2021/2022-Q3); removed: SRE Observability (FY2021/2022-Q2), Goal.Jan 12 2022, 9:46 PM
• lmata edited projects, added: SRE Observability (FY2021/2022-Q4); removed: SRE Observability (FY2021/2022-Q3).Apr 7 2022, 11:03 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2021/2022-Q3) board.
fgiunchedi closed subtask T288726: Move > 60% of observability Prometheus-based checks to Alertmanager as Resolved.Apr 27 2022, 1:21 PM
fgiunchedi edited projects, added: SRE Observability (FY2022/2023-Q1); removed: SRE Observability (FY2021/2022-Q4).Jul 1 2022, 8:18 AM
fgiunchedi added a subtask: T309007: Migrate eventlogging check_prometheus checks to alertmanager.Jul 11 2022, 12:04 PM
fgiunchedi added a subtask: T309009: Migrate eventgate check_prometheus checks to alertmanager.
fgiunchedi added a subtask: T309010: Migrate Kafka prometheus alerts from Icinga to Alertmanager.
fgiunchedi added a subtask: T309011: Migrate labstore prometheus alerts from Icinga to Alertmanager.
fgiunchedi added a subtask: T309012: Migrate zookeeper prometheus checks from Icinga to Alertmanager.
fgiunchedi closed subtask T281454: Onboard teams with Prometheus-based alerts to AlertManager as Resolved.
fgiunchedi added a subtask: T285328: Migrate OSM sync alerts from icinga to AlertManager.Jul 11 2022, 12:08 PM
fgiunchedi added a subtask: T289077: Migrate Search team's prometheus-based alerts from Icinga to alert-manager.
fgiunchedi added a subtask: T293399: Migrate the majority of the analytics cluster alerts from Icinga to AlertManager.
fgiunchedi added a subtask: T294564: Migrate Foundations Prometheus alerts to AlertManager.
fgiunchedi added a subtask: T300723: Migrate Traffic Prometheus alerts from Icinga to Alertmanager.
fgiunchedi reopened subtask T293399: Migrate the majority of the analytics cluster alerts from Icinga to AlertManager as Open.Jul 11 2022, 12:16 PM
gerritbot added a comment.Jul 11 2022, 12:32 PM

Change 812860 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] opensearch: remove icinga::monitor::elasticsearch::old_jvm_gc_checks

https://gerrit.wikimedia.org/r/812860

gerritbot added a project: Patch-For-Review.Jul 11 2022, 12:32 PM
fgiunchedi added a subtask: T311251: Migrate kubernetes alerts away from icinga .Jul 11 2022, 3:50 PM
gerritbot added a comment.Jul 12 2022, 9:30 AM

Change 812860 merged by Filippo Giunchedi:

[operations/puppet@production] opensearch: remove icinga::monitor::elasticsearch::old_jvm_gc_checks

https://gerrit.wikimedia.org/r/812860

Maintenance_bot removed a project: Patch-For-Review.Jul 12 2022, 9:30 AM
BTullis closed subtask T293399: Migrate the majority of the analytics cluster alerts from Icinga to AlertManager as Resolved.Jul 14 2022, 10:17 AM
• lmata moved this task from Inbox to Up next on the SRE Observability (FY2022/2023-Q1) board.Jul 14 2022, 6:09 PM
BCornwall closed subtask T300723: Migrate Traffic Prometheus alerts from Icinga to Alertmanager as Resolved.Jul 27 2022, 4:05 PM
JMeybohm closed subtask T311251: Migrate kubernetes alerts away from icinga as Resolved.Sep 12 2022, 9:52 AM
JMeybohm reopened subtask T311251: Migrate kubernetes alerts away from icinga as Open.Sep 13 2022, 4:46 PM
JMeybohm closed subtask T311251: Migrate kubernetes alerts away from icinga as Resolved.Sep 27 2022, 3:05 PM
fgiunchedi updated the task description. (Show Details)Oct 10 2022, 1:00 PM
fgiunchedi mentioned this in T320931: Progress indicator for Icinga -> Alertmanager migration.Oct 17 2022, 9:06 AM
fgiunchedi added a parent task: T321808: Port all Icinga checks to Prometheus/Alertmanager.Oct 27 2022, 2:03 PM
• lmata edited projects, added: SRE Observability (FY2022/2023-Q2); removed: SRE Observability (FY2022/2023-Q1).Oct 28 2022, 6:30 PM
• lmata moved this task from Inbox to Up next on the SRE Observability (FY2022/2023-Q2) board.
• lmata edited projects, added: SRE Observability (FY2022/2023-Q3); removed: SRE Observability (FY2022/2023-Q2).Jan 11 2023, 10:28 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2022/2023-Q3) board.Jan 12 2023, 12:17 AM
fgiunchedi closed subtask T312764: Port mediawiki prometheus-based alerts from icinga to alertmanager as Resolved.Feb 6 2023, 9:08 AM
BCornwall closed subtask T327791: Move (or delete?) trafficserver restart count alert from icinga to alerts.git as Resolved.Feb 17 2023, 4:38 PM
fgiunchedi closed subtask T309010: Migrate Kafka prometheus alerts from Icinga to Alertmanager as Resolved.Mar 20 2023, 2:43 PM
fgiunchedi closed subtask T327792: Port k8s cache alerts from icinga to alerts.git as Resolved.Mar 21 2023, 1:15 PM
jijiki closed subtask T285328: Migrate OSM sync alerts from icinga to AlertManager as Resolved.Mar 22 2023, 3:21 PM
jbond changed the status of subtask T294564: Migrate Foundations Prometheus alerts to AlertManager from Open to In Progress.Mar 23 2023, 10:52 AM
cmooney closed subtask T327793: Port (or delete) udpmxircecho alerts from icinga to alerts.git as Resolved.Mar 23 2023, 4:37 PM
gerritbot added a comment.Mar 28 2023, 2:22 PM

Change 903654 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] icinga: remove widespread puppet agent alerts

https://gerrit.wikimedia.org/r/903654

gerritbot added a project: Patch-For-Review.Mar 28 2023, 2:22 PM
cmooney closed subtask T309007: Migrate eventlogging check_prometheus checks to alertmanager as Resolved.Mar 28 2023, 8:39 PM
gerritbot added a comment.Mar 29 2023, 1:00 PM

Change 903654 merged by Herron:

[operations/puppet@production] icinga: remove widespread puppet agent alerts

https://gerrit.wikimedia.org/r/903654

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2023, 1:10 PM
jbond closed subtask T294564: Migrate Foundations Prometheus alerts to AlertManager as Resolved.Mar 30 2023, 1:14 PM
fgiunchedi edited projects, added: SRE Observability (FY2022/2023-Q4); removed: SRE Observability (FY2022/2023-Q3).Apr 14 2023, 8:21 AM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2022/2023-Q4) board.May 2 2023, 1:50 PM
herron closed subtask T301944: Web interface to navigate Prometheus alerts and their status as Resolved.May 17 2023, 1:53 PM
fgiunchedi closed subtask T309009: Migrate eventgate check_prometheus checks to alertmanager as Resolved.May 23 2023, 8:20 AM
fgiunchedi closed subtask T312765: Port swift prometheus-based alerts from icinga to alertmanager as Resolved.May 23 2023, 12:13 PM
• lmata edited projects, added: SRE Observability (FY2023/2024-Q1); removed: SRE Observability (FY2022/2023-Q4).Jul 18 2023, 4:56 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q1) board.Jul 18 2023, 8:10 PM
fgiunchedi removed fgiunchedi as the assignee of this task.Sep 19 2023, 8:31 AM
fgiunchedi subscribed.
• lmata edited projects, added: SRE Observability (FY2023/2024-Q2); removed: SRE Observability (FY2023/2024-Q1).Oct 9 2023, 4:19 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q2) board.
• lmata moved this task from Epics In Progress to Inbox on the SRE Observability (FY2023/2024-Q2) board.Nov 29 2023, 3:09 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q2) board.
• lmata edited projects, added: SRE Observability (FY2023/2024-Q3); removed: SRE Observability (FY2023/2024-Q2).Jan 26 2024, 1:05 AM
fgiunchedi edited projects, added: SRE Observability (FY2023/2024-Q4); removed: SRE Observability (FY2023/2024-Q3).Mar 26 2024, 2:57 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q4) board.May 13 2024, 6:35 PM
fgiunchedi edited projects, added: SRE Observability (FY2024/2025-Q1); removed: SRE Observability (FY2023/2024-Q4).Jul 3 2024, 8:10 AM
fgiunchedi removed a subtask: T225140: Icinga alerts that should open tasks instead of alerting.Jul 16 2024, 1:21 PM
fgiunchedi removed a subtask: T305847: Migrate SRE paging alerts off Icinga and to Alertmanager.
fgiunchedi removed a subtask: T301944: Web interface to navigate Prometheus alerts and their status.Jul 16 2024, 1:23 PM
tappof changed the status of subtask T370153: Move kafka-mirror Prometheus-based alerts from Icinga to alerts.git from Open to In Progress.Oct 4 2024, 4:21 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q1) board.Oct 31 2024, 9:26 PM
• lmata edited projects, added: SRE Observability (FY2024/2025-Q2); removed: SRE Observability (FY2024/2025-Q1).Nov 5 2024, 5:04 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q2) board.
• lmata edited projects, added: SRE Observability (FY2024/2025-Q3); removed: SRE Observability (FY2024/2025-Q2).Jan 14 2025, 3:56 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q3) board.
fgiunchedi added a project: Observability-Alerting.Jan 30 2025, 2:29 PM
• lmata edited projects, added: SRE Observability (FY2024/2025-Q4); removed: SRE Observability (FY2024/2025-Q3).Apr 1 2025, 12:04 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q4) board.
• lmata edited projects, added: SRE Observability (FY2025/2026-Q1); removed: SRE Observability (FY2024/2025-Q4).Jun 30 2025, 4:04 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2025/2026-Q1) board.
gerritbot added a comment.Jul 9 2025, 6:47 PM

Change #1167691 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] prometheus: Proof of concept of a nrpe to prometheus translation wrapper

https://gerrit.wikimedia.org/r/1167691

gerritbot added a project: Patch-For-Review.Jul 9 2025, 6:47 PM
tappof closed subtask T309012: Migrate zookeeper prometheus checks from Icinga to Alertmanager as Resolved.Oct 21 2025, 12:26 PM
tappof claimed this task.Oct 21 2025, 12:49 PM
tappof closed subtask T370153: Move kafka-mirror Prometheus-based alerts from Icinga to alerts.git as Resolved.Jul 3 2026, 7:54 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits