Page MenuHomePhabricator
Create Task
Maniphest T288622

All Prometheus based alerts move from Icinga to alert manager exclusively
Open, MediumPublic
Actions

Description

This is a tracking task to move Prometheus-based (i.e. check_prometheus) alerts from Icinga to Alertmanager. These alerts are the most straightforward to port to Prometheus alerting rules since all information by definition is already available in the form of Prometheus metrics.

Details

Other Assignee
lmata
SubjectRepoBranchLines +/-
icinga: remove widespread puppet agent alertsoperations/puppetproduction+0 -25
opensearch: remove icinga::monitor::elasticsearch::old_jvm_gc_checksoperations/puppetproduction+0 -19
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT321808 Port most/all Icinga checks to Prometheus/Alertmanager
 OpenNoneT288622 All Prometheus based alerts move from Icinga to alert manager exclusively
 ResolvedfgiunchediT281454 Onboard teams with Prometheus-based alerts to AlertManager
 ResolvedlmataT281359 Onboard teams with Grafana alerts to AlertManager
 ResolvedfgiunchediT282806 Port traffic/netops grafana alerts to AlertManager
 ResolvedLadsgroupT287741 Convert wikidata-alerts grafana dashboard to AlertManager
 OpenNoneT225140 Icinga alerts that should open tasks instead of alerting
 ResolvedfgiunchediT310266 Move mgmt SSH checks from Icinga to Prometheus/Alertmanager
 InvalidNoneT319299 Investigate longer run time for hiera_export netbox script
 ResolvedVolansT320721 Decide whether decom'ing hosts mgmt DNS entry should stay or not
 ResolvedfgiunchediT288726 Move > 60% of observability Prometheus-based checks to Alertmanager
 ResolvedherronT301944 Web interface to navigate Prometheus alerts and their status
 OpenherronT326657 Add prometheus-https load balancer
 OpenNoneT305847 Migrate SRE paging alerts off Icinga and to Alertmanager
 ResolvedfgiunchediT309546 Export etcdmirror 'lag' metric and alert on it
 ResolvedNoneT309007 Migrate eventlogging check_prometheus checks to alertmanager
 ResolvedArnoldokothT309009 Migrate eventgate check_prometheus checks to alertmanager
 ResolvedfgiunchediT309010 Migrate Kafka prometheus alerts from Icinga to Alertmanager
 DuplicateNoneT309011 Migrate labstore prometheus alerts from Icinga to Alertmanager
 OpenNoneT309012 Migrate zookeeper prometheus checks from Icinga to Alertmanager
 ResolvedNoneT285328 Migrate OSM sync alerts from icinga to AlertManager
 ResolvedEBernhardsonT289077 Migrate Search team's prometheus-based alerts from Icinga to alert-manager
 ResolvedBTullisT293399 Migrate the majority of the analytics cluster alerts from Icinga to AlertManager
 ResolvedjbondT294564 Migrate Foundations Prometheus alerts to AlertManager
 In ProgressBUG REPORTNoneT302639 How should we monitor for faulty memory modules?
 ResolvedjhathawayT302687 Don't move filesystem_avail_bigger_than_size icinga check to alert manager?
 ResolvedBCornwallT300723 Migrate Traffic Prometheus alerts from Icinga to Alertmanager
 ResolvedBCornwallT311445 Create vm.max_map_count metrics for Prometheus
 DuplicateNoneT312763 Port kubernetes prometheus-based alerts from icinga to alertmanager
 ResolvedfgiunchediT312764 Port mediawiki prometheus-based alerts from icinga to alertmanager
 ResolvedfgiunchediT312765 Port swift prometheus-based alerts from icinga to alertmanager
 ResolvedJMeybohmT311251 Migrate kubernetes alerts away from icinga
ResolvedBCornwallT327791 Move (or delete?) trafficserver restart count alert from icinga to alerts.git
 ResolvedfgiunchediT327792 Port k8s cache alerts from icinga to alerts.git
 ResolvedcmooneyT327793 Port (or delete) udpmxircecho alerts from icinga to alerts.git

Event Timeline

lmata created this task.Aug 11 2021, 1:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 11 2021, 1:21 PM
lmata added a project: Goal.Aug 11 2021, 1:21 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2021/2022-Q1) board.Aug 11 2021, 1:24 PM
lmata updated Other Assignee, added: lmata.Aug 11 2021, 1:28 PM
fgiunchedi added subtasks: T281454: Onboard teams with Prometheus-based alerts to AlertManager, T281359: Onboard teams with Grafana alerts to AlertManager, T225140: Icinga alerts that should open tasks instead of alerting.Aug 12 2021, 9:00 AM
lmata awarded a token.Aug 12 2021, 6:32 PM
RKemper removed a subtask: T289077: Migrate Search team's prometheus-based alerts from Icinga to alert-manager.Aug 17 2021, 6:47 PM
fgiunchedi closed subtask T281359: Onboard teams with Grafana alerts to AlertManager as Resolved.Sep 28 2021, 11:48 AM
lmata triaged this task as Medium priority.Sep 30 2021, 9:51 PM
lmata edited projects, added SRE Observability (FY2021/2022-Q2); removed SRE Observability (FY2021/2022-Q1).
lmata updated the task description. (Show Details)
lmata edited projects, added SRE Observability (FY2021/2022-Q3); removed SRE Observability (FY2021/2022-Q2), Goal.Jan 12 2022, 9:46 PM
lmata edited projects, added SRE Observability (FY2021/2022-Q4); removed SRE Observability (FY2021/2022-Q3).Apr 7 2022, 11:03 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2021/2022-Q3) board.
fgiunchedi closed subtask T288726: Move > 60% of observability Prometheus-based checks to Alertmanager as Resolved.Apr 27 2022, 1:21 PM
fgiunchedi edited projects, added SRE Observability (FY2022/2023-Q1); removed SRE Observability (FY2021/2022-Q4).Jul 1 2022, 8:18 AM
fgiunchedi added a subtask: T309007: Migrate eventlogging check_prometheus checks to alertmanager.Jul 11 2022, 12:04 PM
fgiunchedi added a subtask: T309009: Migrate eventgate check_prometheus checks to alertmanager.
fgiunchedi added a subtask: T309010: Migrate Kafka prometheus alerts from Icinga to Alertmanager.
fgiunchedi added a subtask: T309011: Migrate labstore prometheus alerts from Icinga to Alertmanager.
fgiunchedi added a subtask: T309012: Migrate zookeeper prometheus checks from Icinga to Alertmanager.
fgiunchedi closed subtask T281454: Onboard teams with Prometheus-based alerts to AlertManager as Resolved.
fgiunchedi added a subtask: T285328: Migrate OSM sync alerts from icinga to AlertManager.Jul 11 2022, 12:08 PM
fgiunchedi added a subtask: T289077: Migrate Search team's prometheus-based alerts from Icinga to alert-manager.
fgiunchedi added a subtask: T293399: Migrate the majority of the analytics cluster alerts from Icinga to AlertManager.
fgiunchedi added a subtask: T294564: Migrate Foundations Prometheus alerts to AlertManager.
fgiunchedi added a subtask: T300723: Migrate Traffic Prometheus alerts from Icinga to Alertmanager.
fgiunchedi reopened subtask T293399: Migrate the majority of the analytics cluster alerts from Icinga to AlertManager as Open.Jul 11 2022, 12:16 PM
gerritbot added a comment.Jul 11 2022, 12:32 PM

Change 812860 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] opensearch: remove icinga::monitor::elasticsearch::old_jvm_gc_checks

https://gerrit.wikimedia.org/r/812860

gerritbot added a project: Patch-For-Review.Jul 11 2022, 12:32 PM
fgiunchedi added a subtask: T311251: Migrate kubernetes alerts away from icinga .Jul 11 2022, 3:50 PM
gerritbot added a comment.Jul 12 2022, 9:30 AM

Change 812860 merged by Filippo Giunchedi:

[operations/puppet@production] opensearch: remove icinga::monitor::elasticsearch::old_jvm_gc_checks

https://gerrit.wikimedia.org/r/812860

Maintenance_bot removed a project: Patch-For-Review.Jul 12 2022, 9:30 AM
BTullis closed subtask T293399: Migrate the majority of the analytics cluster alerts from Icinga to AlertManager as Resolved.Jul 14 2022, 10:17 AM
lmata moved this task from Inbox to Up next on the SRE Observability (FY2022/2023-Q1) board.Jul 14 2022, 6:09 PM
BCornwall closed subtask T300723: Migrate Traffic Prometheus alerts from Icinga to Alertmanager as Resolved.Jul 27 2022, 4:05 PM
JMeybohm closed subtask T311251: Migrate kubernetes alerts away from icinga as Resolved.Sep 12 2022, 9:52 AM
JMeybohm reopened subtask T311251: Migrate kubernetes alerts away from icinga as Open.Sep 13 2022, 4:46 PM
JMeybohm closed subtask T311251: Migrate kubernetes alerts away from icinga as Resolved.Sep 27 2022, 3:05 PM
fgiunchedi updated the task description. (Show Details)Oct 10 2022, 1:00 PM
fgiunchedi mentioned this in T320931: Progress indicator for Icinga -> Alertmanager migration.Oct 17 2022, 9:06 AM
fgiunchedi added a parent task: T321808: Port most/all Icinga checks to Prometheus/Alertmanager.Oct 27 2022, 2:03 PM
lmata edited projects, added SRE Observability (FY2022/2023-Q2); removed SRE Observability (FY2022/2023-Q1).Oct 28 2022, 6:30 PM
lmata moved this task from Inbox to Up next on the SRE Observability (FY2022/2023-Q2) board.
lmata edited projects, added SRE Observability (FY2022/2023-Q3); removed SRE Observability (FY2022/2023-Q2).Jan 11 2023, 10:28 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2022/2023-Q3) board.Jan 12 2023, 12:17 AM
fgiunchedi closed subtask T312764: Port mediawiki prometheus-based alerts from icinga to alertmanager as Resolved.Feb 6 2023, 9:08 AM
BCornwall closed subtask T327791: Move (or delete?) trafficserver restart count alert from icinga to alerts.git as Resolved.Feb 17 2023, 4:38 PM
fgiunchedi closed subtask T309010: Migrate Kafka prometheus alerts from Icinga to Alertmanager as Resolved.Mar 20 2023, 2:43 PM
fgiunchedi closed subtask T327792: Port k8s cache alerts from icinga to alerts.git as Resolved.Mar 21 2023, 1:15 PM
jijiki closed subtask T285328: Migrate OSM sync alerts from icinga to AlertManager as Resolved.Mar 22 2023, 3:21 PM
jbond changed the status of subtask T294564: Migrate Foundations Prometheus alerts to AlertManager from Open to In Progress.Mar 23 2023, 10:52 AM
cmooney closed subtask T327793: Port (or delete) udpmxircecho alerts from icinga to alerts.git as Resolved.Mar 23 2023, 4:37 PM
gerritbot added a comment.Mar 28 2023, 2:22 PM

Change 903654 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] icinga: remove widespread puppet agent alerts

https://gerrit.wikimedia.org/r/903654

gerritbot added a project: Patch-For-Review.Mar 28 2023, 2:22 PM
cmooney closed subtask T309007: Migrate eventlogging check_prometheus checks to alertmanager as Resolved.Mar 28 2023, 8:39 PM
gerritbot added a comment.Mar 29 2023, 1:00 PM

Change 903654 merged by Herron:

[operations/puppet@production] icinga: remove widespread puppet agent alerts

https://gerrit.wikimedia.org/r/903654

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2023, 1:10 PM
jbond closed subtask T294564: Migrate Foundations Prometheus alerts to AlertManager as Resolved.Mar 30 2023, 1:14 PM
fgiunchedi edited projects, added SRE Observability (FY2022/2023-Q4); removed SRE Observability (FY2022/2023-Q3).Apr 14 2023, 8:21 AM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2022/2023-Q4) board.May 2 2023, 1:50 PM
herron closed subtask T301944: Web interface to navigate Prometheus alerts and their status as Resolved.May 17 2023, 1:53 PM
fgiunchedi closed subtask T309009: Migrate eventgate check_prometheus checks to alertmanager as Resolved.May 23 2023, 8:20 AM
fgiunchedi closed subtask T312765: Port swift prometheus-based alerts from icinga to alertmanager as Resolved.May 23 2023, 12:13 PM
lmata edited projects, added SRE Observability (FY2023/2024-Q1); removed SRE Observability (FY2022/2023-Q4).Jul 18 2023, 4:56 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q1) board.Jul 18 2023, 8:10 PM
fgiunchedi removed fgiunchedi as the assignee of this task.Sep 19 2023, 8:31 AM
fgiunchedi subscribed.
lmata edited projects, added SRE Observability (FY2023/2024-Q2); removed SRE Observability (FY2023/2024-Q1).Oct 9 2023, 4:19 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q2) board.
lmata moved this task from Epics In Progress to Inbox on the SRE Observability (FY2023/2024-Q2) board.Nov 29 2023, 3:09 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q2) board.
lmata edited projects, added SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).Jan 26 2024, 1:05 AM
fgiunchedi edited projects, added SRE Observability (FY2023/2024-Q4); removed SRE Observability (FY2023/2024-Q3).Mar 26 2024, 2:57 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q4) board.May 13 2024, 6:35 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits