Port all Icinga checks to Prometheus/Alertmanager: preparation
Open, Needs TriagePublic
Actions

Assigned To

Authored By

	tappof
	May 28 2025, 12:52 PM

Project Tags

Referenced Files

None

Subscribers

Description

This is a tracking task for the overall work related to the preparation steps for migrating alerts from Icinga to Prometheus/Alertmanager.

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		tappof	T395441 Port all Icinga checks to Prometheus/Alertmanager: preparation
Open		tappof	T321808 Port all Icinga checks to Prometheus/Alertmanager
Open		tappof	T288622 All Prometheus based alerts move from Icinga to alert manager exclusively
Resolved		fgiunchedi	T281454 Onboard teams with Prometheus-based alerts to AlertManager
Resolved		• lmata	T281359 Onboard teams with Grafana alerts to AlertManager
Resolved		fgiunchedi	T282806 Port traffic/netops grafana alerts to AlertManager
Resolved		fgiunchedi	T288726 Move > 60% of observability Prometheus-based checks to Alertmanager
Resolved		None	T309007 Migrate eventlogging check_prometheus checks to alertmanager
Resolved		Arnoldokoth	T309009 Migrate eventgate check_prometheus checks to alertmanager
Resolved		fgiunchedi	T309010 Migrate Kafka prometheus alerts from Icinga to Alertmanager
Duplicate		None	T309011 Migrate labstore prometheus alerts from Icinga to Alertmanager
Resolved	Goal	tappof	T309012 Migrate zookeeper prometheus checks from Icinga to Alertmanager
Resolved		None	T285328 Migrate OSM sync alerts from icinga to AlertManager
Resolved		EBernhardson	T289077 Migrate Search team's prometheus-based alerts from Icinga to alert-manager
Resolved		BTullis	T293399 Migrate the majority of the analytics cluster alerts from Icinga to AlertManager
Resolved		jbond	T294564 Migrate Foundations Prometheus alerts to AlertManager
Resolved		jhathaway	T302687 Don't move filesystem_avail_bigger_than_size icinga check to alert manager?
Resolved		BCornwall	T300723 Migrate Traffic Prometheus alerts from Icinga to Alertmanager
Resolved		BCornwall	T311445 Create vm.max_map_count metrics for Prometheus
Duplicate		None	T312763 Port kubernetes prometheus-based alerts from icinga to alertmanager
Resolved		fgiunchedi	T312764 Port mediawiki prometheus-based alerts from icinga to alertmanager
Resolved		fgiunchedi	T312765 Port swift prometheus-based alerts from icinga to alertmanager
Resolved		JMeybohm	T311251 Migrate kubernetes alerts away from icinga
Resolved		BCornwall	T327791 Move (or delete?) trafficserver restart count alert from icinga to alerts.git
Resolved		fgiunchedi	T327792 Port k8s cache alerts from icinga to alerts.git
Resolved		cmooney	T327793 Port (or delete) udpmxircecho alerts from icinga to alerts.git
Resolved	Goal	tappof	T370153 Move kafka-mirror Prometheus-based alerts from Icinga to alerts.git
Resolved		tappof	T381665 module to define custom Prometheus alerts directly in Puppet
Resolved		tappof	T425621 Move Kafka mirror monitors and alerts to the alerts repo
Invalid		None	T321810 Move systemd unit status checks to Alertmanager
Open	Goal	None	T315866 Migrate mysql icinga alerts to alert manager
Resolved		ABran-WMF	T367278 Migrate mysql icinga alerts to alert manager - pt-heartbeat + scaffolding
Resolved		BTullis	T371049 prometheus-mysqld-exporter doesn't fully support multi-instances for pt-heartbeat
Resolved		BTullis	T376978 Alert in need of triage: PrometheusMysqldExporterFailed (instance db1208:13351)
Resolved		ABran-WMF	T367279 Migrate mysql icinga alerts to alert manager - seconds_behind_master + threads (replication/io)
Declined		ABran-WMF	T367280 Migrate mysql icinga alerts to alert manager - memory pressure
Resolved		ABran-WMF	T367281 Migrate mysql icinga alerts to alert manager - disk pressure
Open		None	T367282 Migrate mysql icinga alerts to alert manager - read only status
Open		None	T367283 Migrate mysql icinga alerts to alert manager - process monitoring
Open		None	T367284 Migrate mysql icinga alerts to alert manager - mariadb errors
In Progress		None	T369045 Migrate mysql icinga alerts to alert manager - haproxy exporter
Resolved		fgiunchedi	T332709 Multi-team Prometheus/Alertmanager alerts
Resolved		jbond	T349176 Route systemd unit alerts to the correct team
Resolved	Goal	colewhite	T332764 Port base host checks from Icinga to Alertmanager
Resolved		cmooney	T333007 validate what we need from the check_eth check
Resolved		SLyngshede-WMF	T347312 Expose ethtool metrics to Prometheus
Resolved		SLyngshede-WMF	T351068 ethtool-exporter collection/parsing errors
Resolved		hnowlan	T353323 Improve the InterfaceSpeedError alert
Resolved		fgiunchedi	T337831 Remove specific nrpe::monitor_systemd_unit_state
Resolved		fgiunchedi	T343902 Port "Ensure hosts are not performing a change on every puppet run"
Open		None	T350357 Make it easier to export textfiles for node-exporter from puppet
Open	Goal	SLyngshede-WMF	T350694 Infrastructure Foundation Alerts to migrate
Resolved	BUG REPORT	fgiunchedi	T353457 Karma UI shows duplicate alerts
Open	Goal	• herron	T357099 Remove check_procs-based Icinga alerts
Open	Goal	SLyngshede-WMF	T367065 Move profile::idp::client::httpd::site checks to Prometheus blackbox probes
Open	Goal	• herron	T367149 Add "file age" node textfile exporter capability
Open		None	T225140 Icinga alerts that should open tasks instead of alerting
Resolved		fgiunchedi	T310266 Move mgmt SSH checks from Icinga to Prometheus/Alertmanager
Invalid		None	T319299 Investigate longer run time for hiera_export netbox script
Resolved		Volans	T320721 Decide whether decom'ing hosts mgmt DNS entry should stay or not
Resolved		fgiunchedi	T305847 Migrate SRE paging alerts off Icinga and to Alertmanager
Resolved		fgiunchedi	T309546 Export etcdmirror 'lag' metric and alert on it
Open	Goal	• herron	T370157 Port lists monitoring alerts to Alertmanager
Resolved		tappof	T370506 Replace check_ripe_atlas with prometheus alert
Resolved	BUG REPORT	tappof	T381561 deployment site for prometheus::blackbox::check::(icmp\|http\|tcp) is not driven by the $site parameter
Resolved		tappof	T381580 cloudgw ICMP checks are affected by changes to prometheus::blackbox::check::icmp
Resolved		cmooney	T388379 ProbeDown
Resolved		tappof	T388419 cannot ping ripe-atlas-codfw from the codfw prometheus instance
Resolved		tappof	T388801 Missing atlas-esams device in netbox
Open	Goal	tappof	T370526 Remove load_average check for ms-be/thanos-be
Resolved		andrea.denisse	T370527 Remove mediawiki-installation dsh group check
Open	Goal	tappof	T370530 Clean up "git repo needs merge" checks
Resolved		• Stevemunene	T371080 Port disk space check for hadoop worker to Alertmanager
Open	Goal	tappof	T371083 Port profile::opensearch::monitoring::base_checks to Prometheus/Alertmanager
Resolved	BUG REPORT	tappof	T370680 Unassigned shard email alerts are bouncing, because the status line is too long for an email subject
Resolved		tappof	T373510 opensearch general purpose dashboard
Resolved		BTullis	T374821 Replace or delete dumps_store_load_average
In Progress	Goal	SLyngshede-WMF	T374823 Port netbox reports checks to Prometheus/Alertmanager
Open		SLyngshede-WMF	T374827 Port conntrack utilization alert to Prometheus/Alertmanager
Open	Goal	• herron	T374839 Port postgresql replication check to Prometheus/Alertmanager
Resolved		fgiunchedi	T374842 Retire anycast_healthchecker Icinga check
Resolved		taavi	T375082 Upgrade cloudlb hosts to bookworm
Resolved		fgiunchedi	T374860 Retire mw_wikiversion_difference check
Resolved		BTullis	T374916 Port Categories lag / ping checks to Prometheus/Alertmanager
Open	Goal	tappof	T375166 Port PDU checks to Prometheus/Alertmanager
Resolved		fgiunchedi	T375271 Remove uwsgi::app unit monitoring
Resolved	Goal	colewhite	T384214 Port or deprecate Icinga airflow checks
Open	Goal	None	T384303 Port or deprecate legal html checks
Open	Goal	None	T384305 Port or deprecate memcached nrpe to Prometheus/Alertmanager
Open	Goal	None	T384308 Port or deprecate nrpe blazegraph checks
Open	Goal	None	T384309 Port or remove cloudelastic icinga checks
Open	Goal	None	T384321 Port or delete docker registry icinga checks
Resolved		fgiunchedi	T384324 Port or delete chartmuseum icinga checks
Open	Goal	None	T384425 Port DNS icinga checks to Alertmanager
Open	Goal	None	T384427 Port or remove eventstreams icinga check
Open	Goal	None	T384438 Port or deprecate wikidough icinga checks
Open	Goal	None	T384571 Move check_bacula to Prometheus/Alertmanager
Open	Goal	None	T384830 Port network checks in netops module to Prometheus/Alertmanager
Open	Goal	None	T384922 Port gitlab and gerrit monitoring to Prometheus/Alertmanager
Resolved	Goal	hnowlan	T384924 Port or remove redis checks
Open	Goal	None	T384933 Port or deprecate etcd Icinga checks to Prometheus/Alertmanager
Open	Goal	None	T384938 Port or deprecate pybal Icinga checks
Open	Goal	None	T384939 Port zuul checks to Prometheus/Alertmanager
Open	Goal	None	T384998 Port elasticsearch checks from Icinga to Prometheus/Alertmanager
Open	Goal	None	T385583 Migrate swift icinga alerts to Prometheus/Alertmanager
Open	Goal	None	T385587 Port frontend CDN icinga checks to prometheus/alertmanager
Open	Goal	None	T385590 Port ntp monitoring to Prometheus/Alertmanager
Open	Goal	bking	T358029 Migrate selected Search Platform alerts from icinga search-platform team to prometheus data-platform team
Open	Goal	None	T328502 Move WMCS off of Icinga and introduce alertmanager
Resolved		fgiunchedi	T345294 Move Cloud VPS control plane alerting to alertmanager
Resolved		fgiunchedi	T345983 Remove Icinga checks for Cloud VPS projects (not: infrastructure)
Resolved		fgiunchedi	T313030 [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones
Resolved		taavi	T311911 Upgrade Toolforge Prometheus servers to Debian Bullseye or later
Resolved		• nskaggs	T312692 Request increased quota for tools Cloud VPS project
Resolved		taavi	T284860 Prometheus alerting support on Toolforge
Resolved		taavi	T304716 Cloud services enhancement proposal: Prometheus metrics for Toolforge/Toolsbeta/Paws Kubernetes clusters
Resolved		dcaro	T313031 Paging for alerts from the cloud realm
Resolved		taavi	T358333 Remove toolschecker grid engine checks
Open	Goal	None	T362397 Move wikitech-static monitoring off Icinga
Open		None	T429451 Establish a blackbox network probe vantage point into cloud realm
Open	Goal	None	T407117 Move remaining TCP/HTTP/HTTPS checks to blackbox
Open	Goal	None	T407120 O11y alerts to migrate
Open	Goal	None	T407130 Remove check_systemd_unit_status-based Icinga alerts
Open	Goal	None	T407137 Port haproxy checks to prometheus/alertmanager
Open	Goal	None	T407138 Port hadoop checks to prometheus/alertmanager
Resolved	Goal	None	T407141 Port restbase checks to prometheus/alertmanager
Open	Goal	None	T407329 Port orchestrator checks to Prometheus/Alertmanager
Open	Goal	None	T407330 Port lvs checks to Prometheus/Alertmanager
Open	Goal	None	T407331 Port mediawiki::php checks to Prometheus/Alertmanager
Resolved		tappof	T395442 Setup reliable migration dashboards
Open	Goal	None	T395445 Migrate away from Icinga service checks that can trigger pages (non-db only)
Open		None	T395447 Prototype / experiment with moving raid checks to alertmanager
Open	Goal	None	T395448 Discuss about "host down" semantics
Open		None	T395449 Reimage cookbook icinga logic review
Open		tappof	T350360 Evaluate "drop in" replacement for nrpe scripts
Open	Goal	tappof	T384472 Candidate nrpe checks for compatibility layer icinga/prometheus/alertmanager
Open		tappof	T413743 Alert in need of triage: nrpe_Check_whether_ferm_is_active_by_checking_the_default_input_chain (instance cumin2002:9100)
Open		tappof	T413744 Alert in need of triage: nrpe_Check_whether_ferm_is_active_by_checking_the_default_input_chain (instance cloudidp2001-dev:9100)
Open		tappof	T413745 Alert in need of triage: nrpe_Check_whether_ferm_is_active_by_checking_the_default_input_chain (instance cloudcumin2001:9100)
Open		tappof	T384837 Integration between alertmanager and cumin cookbooks
Open		tappof	T395446 Evaluate which solution we could adopt as a drop-in replacement for NRPE (and start prototyping)
Open		tappof	T376906 Check how LibreNMS sends out alerts.
Resolved		tappof	T388506 Implement a less noisy way to remove nrpe checks (without UNKNOWN spam)
Resolved		tappof	T387866 Configure PDU monitoring resources through NetBox
Open		tappof	T388138 PDUs in Active status missing IP address information in NetBox
Resolved		fgiunchedi	T333638 Icinga deprecation and sunsetting roadmap
Resolved		fgiunchedi	T320931 Progress indicator for Icinga -> Alertmanager migration
Resolved		tappof	T395443 Automate Snapshot Generation for Icinga to Prometheus/Alermanager Migration
Resolved		Ladsgroup	T287741 Convert wikidata-alerts grafana dashboard to AlertManager
Open		None	T330989 Probe mr devices from Prometheus
Resolved		Dwisehaupt	T367370 Shift frack alerting to use prometheus-alertmanager instead of icinga
Resolved		Dwisehaupt	T411036 Replace pay-lb anycast-healthcheck check_cmd with a non-nagios alternative
Open		None	T415604 create cli tooling to handle alert/silence management for frack prometheus based alerts
Resolved		Dwisehaupt	T422888 Update firewall rules to allow frtech hosts to send alerts to production alertmanger