Move WMCS off of Icinga and introduce alertmanager
Closed, ResolvedPublicGoal
Actions

Assigned To

Authored By

	KHernandez-WMF
	Jan 31 2023, 11:09 PM

Project Tags

Referenced Files

None

Subscribers

View All 13 Subscribers

Description

Below is a list of icinga checks with contact group wmcs that will need to be migrated to AM/prometheus (or deprecated) for wmcs to be off icinga (modulo base checks, out of scope for this)

! MIGRATION TABLE !

Migrated? (Y/N)	Deleted? (Y/N)	Title	Resource Type	Command	File	Profiles
N	Y	Auth DNS TCP: X.eqiad1.X on server X.openstack.eqiad1.wikimediacloud.org	Monitoring::Service	check_dig_tcp	modules/profile/manifests/openstack/eqiad1/pdns/auth/service.pp:32	profile::openstack::eqiad1::pdns::auth::service
N	Y	Auth DNS UDP: X.eqiad1.X on server X.openstack.eqiad1.wikimediacloud.org	Monitoring::Service	check_dig	modules/profile/manifests/openstack/eqiad1/pdns/auth/service.pp:25	profile::openstack::eqiad1::pdns::auth::service
N	Y	check-flavor_aggregates	Nrpe::Monitor_service	/usr/local/lib/nagios/plugins/check_flavor_properties	modules/openstack/manifests/nova/fullstack/monitor.pp:8	profile::openstack::base::nova::fullstack::service, profile::openstack::eqiad1::nova::fullstack::service, profile::openstack::codfw1dev::nova::fullstack::service
N	Y	ensure_running_kvm_instances	Nrpe::Monitor_service	/usr/lib/nagios/plugins/check_procs	modules/openstack/manifests/nova/compute/monitor.pp:32	profile::openstack::eqiad1::nova::compute::service
N	Y	check-cinder-snapshot-leaks	Nrpe::Monitor_service	/usr/local/bin/check_cinder_snapshot_leaks.py	modules/openstack/manifests/cinder/monitor.pp:17	profile::openstack::base::cinder, profile::openstack::codfw1dev::cinder, profile::openstack::eqiad1::cinder
N	Y	check-neutron-conntrack	Nrpe::Monitor_service	/usr/local/lib/nagios/plugins/check_neutron_conntrack	modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp:7	profile::openstack::base::neutron::l3_agent, profile::openstack::eqiad1::neutron::l3_agent, profile::openstack::codfw1dev::neutron::l3_agent
N	Y	tools-checker-redis	Monitoring::Service	check_http_url_at_address_for_string_with_timeout	modules/icinga/manifests/monitor/toollabs.pp:61	profile::icinga
N	Y	tools-checker-etcd-k8s	Monitoring::Service	check_http_url_at_address_for_string_with_timeout	modules/icinga/manifests/monitor/toollabs.pp:23	profile::icinga
N	Y	tools-checker-nfs-home	Monitoring::Service	check_http_url_at_address_for_string_with_timeout	modules/icinga/manifests/monitor/toollabs.pp:52	profile::icinga
N	Y	tools-checker-labs-dns-private	Monitoring::Service	check_http_url_at_address_for_string_with_timeout	modules/icinga/manifests/monitor/toollabs.pp:14	profile::icinga
N	Y	tools-checker-dumps	Monitoring::Service	check_http_url_at_address_for_string_with_timeout	modules/icinga/manifests/monitor/toollabs.pp:43	profile::icinga
N	Y	tools-checker-ldap	Monitoring::Service	check_http_url_at_address_for_string_with_timeout	modules/icinga/manifests/monitor/toollabs.pp:34	profile::icinga
N	Y	tools-checker-self	Monitoring::Service	check_http_url_at_address_for_string_with_timeout	modules/icinga/manifests/monitor/toollabs.pp:70	profile::icinga

Filippo's observations

auth dns checks likely will require the equivalent of prometheus::blackbox::check::tcp for dns checks
check-flavor_aggregates ?
ensure_running_kvm_instances should be removed in favor of higher level service (e.g. nova) signals
check-cinder-snapshot-leaks ?
check-neutron-conntrack needs adapting to export prometheus metrics instead via node-exporter textfile
tools-checker-* is tackled as part of T313030: [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
openldap: stop using icinga in clouddev	Filippo Giunchedi	operations/puppet	production	+0 -8
openstack: stop using icinga for pdns auth checks	Filippo Giunchedi	operations/puppet	production	+0 -29
openstack: remove openstack::monitor::neutron::l3_agent_conntrack	Filippo Giunchedi	operations/puppet	production	+0 -124
openstack: deprecate icinga check-neutron-conntrack	Filippo Giunchedi	operations/puppet	production	+1 -1
Disable Memcached Icinga checks for cloudcontrols/cloudwebs	Majavah	operations/puppet	production	+7 -4
team-wmcs: Port Memcached availability alert	Majavah	operations/alerts	master	+46 -0
team-wmcs: introduce per-namespace neutron conntrack alert	Filippo Giunchedi	operations/alerts	master	+56 -0
openstack: deprecate icinga check-flavor_aggregates	Filippo Giunchedi	operations/puppet	production	+0 -145
openstack: deprecate check-cinder-snapshot-leaks	Filippo Giunchedi	operations/puppet	production	+0 -73
openstack: deprecate ensure_running_kvm_instances check	Filippo Giunchedi	operations/puppet	production	+0 -49
team-wmcs: do not pint-warn on NeutronAgentAdminDown	Filippo Giunchedi	operations/alerts	master	+1 -0
prometheus: remove 'key' label from neutron_netns metrics	Filippo Giunchedi	operations/puppet	production	+2 -2
nova: enable nrpe2nodexp wrapper on check-flavor_aggregates	Tiziano Fogli	operations/puppet	production	+10 -9
neutron: enable nrpe2nodexp wrapper on check-neutron-conntrack	Tiziano Fogli	operations/puppet	production	+8 -7
cinder: enable nrpe2nodexp wrapper on check-cinder-snapshot-leaks	Tiziano Fogli	operations/puppet	production	+8 -7
wmcs: remove HighIOWaitStalling	Filippo Giunchedi	operations/alerts	master	+0 -34
wmcs: Migrate network saturation alerts to the alerts.git repository	Andrea Denisse	operations/alerts	master	+66 -0
wmcs: Remove Puppet files for migrated Prometheus alerts	Andrea Denisse	operations/puppet	production	+0 -74
wmcs: Migrate iowait stalling alerts to the alerts.git repository	Andrea Denisse	operations/alerts	master	+34 -0

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		tappof	T395441 Port all Icinga checks to Prometheus/Alertmanager: preparation
Open		tappof	T321808 Port all Icinga checks to Prometheus/Alertmanager
Resolved	Goal	fgiunchedi	T328502 Move WMCS off of Icinga and introduce alertmanager
Resolved		fgiunchedi	T345294 Move Cloud VPS control plane alerting to alertmanager
Resolved		fgiunchedi	T345983 Remove Icinga checks for Cloud VPS projects (not: infrastructure)
Resolved		fgiunchedi	T313030 [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones
Resolved		• taavi	T311911 Upgrade Toolforge Prometheus servers to Debian Bullseye or later
Resolved		• nskaggs	T312692 Request increased quota for tools Cloud VPS project
Resolved		• taavi	T284860 Prometheus alerting support on Toolforge
Resolved		• taavi	T304716 Cloud services enhancement proposal: Prometheus metrics for Toolforge/Toolsbeta/Paws Kubernetes clusters
Resolved		dcaro	T313031 Paging for alerts from the cloud realm
Resolved		• taavi	T358333 Remove toolschecker grid engine checks
Resolved	Goal	fgiunchedi	T362397 Move wikitech-static monitoring off Icinga
Open		None	T429451 Establish a blackbox network probe vantage point into cloud realm

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2023/2024-Q3-Q4) board.Feb 1 2024, 11:13 AM

In case it is useful, as part of the icinga migration I've been collecting checks and their type/status here: https://docs.google.com/spreadsheets/d/19nxCXldb804TJCXGy4Z2BHG_1wRksRnKcPC6sXfjQuM/edit#gid=1831147731

dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.Feb 22 2024, 3:00 PM

• taavi removed • taavi as the assignee of this task.Jun 25 2024, 3:35 PM

A very low hanging fruit to make progress on this task is the following prometheus-based checks:

monitoring::check_prometheus { 'network_out_saturated':
monitoring::check_prometheus { 'network_in_saturated':
monitoring::check_prometheus { 'high_iowait_stalling':

Those are straightforward to port to alerts.git since they are already Prometheus-based, or even better deleted if they are no longer relevant

fnegri edited projects, added: cloud-services-team (FY2024/2025-Q1-Q2); removed: cloud-services-team (FY2023/2024-Q3-Q4).Aug 19 2024, 9:18 AM

fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2024/2025-Q1-Q2) board.

andrea.denisse subscribed.Jan 9 2025, 11:12 PM

fnegri edited projects, added: cloud-services-team (FY2024/2025-Q3-Q4); removed: cloud-services-team (FY2024/2025-Q1-Q2).Jan 14 2025, 2:36 PM

fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2024/2025-Q3-Q4) board.

Change #1111328 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/alerts@master] wmcs: Migrate network saturation alerts to the alerts.git repository

https://gerrit.wikimedia.org/r/1111328

gerritbot added a project: Patch-For-Review.Jan 14 2025, 8:58 PM

Change #1111338 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/alerts@master] wmcs: Migrate iowait stalling alerts to the alerts.git repository

https://gerrit.wikimedia.org/r/1111338

Change #1111340 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] wmcs: Remove Puppet files for migrated Prometheus alerts

https://gerrit.wikimedia.org/r/1111340

Change #1111338 merged by Andrea Denisse:

[operations/alerts@master] wmcs: Migrate iowait stalling alerts to the alerts.git repository

https://gerrit.wikimedia.org/r/1111338

Change #1111328 merged by Andrea Denisse:

[operations/alerts@master] wmcs: Migrate network saturation alerts to the alerts.git repository

https://gerrit.wikimedia.org/r/1111328

Change #1111340 merged by Andrea Denisse:

[operations/puppet@production] wmcs: Remove Puppet files for migrated Prometheus alerts

https://gerrit.wikimedia.org/r/1111340

andrea.denisse updated the task description. (Show Details)Jan 21 2025, 7:51 PM

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2025, 8:30 PM

fgiunchedi moved this task from Inbox to Radar on the Observability-Alerting board.Mar 20 2025, 1:21 PM

Resetting task status from "In Progress" to "Open" as this task has been "in progress" for more than one year (see T380300). Feel free to set that status again, or rather break down into smaller subtasks.

Change #1155138 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] monitoring services: add migration task T328502 to instances

https://gerrit.wikimedia.org/r/1155138

gerritbot added a project: Patch-For-Review.Jun 10 2025, 8:54 AM

Change #1155138 merged by Tiziano Fogli:

[operations/puppet@production] monitoring services: add migration task T328502 to instances

https://gerrit.wikimedia.org/r/1155138

Maintenance_bot removed a project: Patch-For-Review.Jun 23 2025, 12:32 PM

fnegri edited projects, added: Epic; removed: cloud-services-team (FY2024/2025-Q3-Q4).Jul 15 2025, 1:33 PM

Herald added a project: cloud-services-team. · View Herald TranscriptJul 15 2025, 1:33 PM

fgiunchedi updated the task description. (Show Details)Sep 1 2025, 1:13 PM

tappof changed the subtype of this task from "Task" to "Goal".Sep 2 2025, 1:33 PM

tappof updated the task description. (Show Details)Sep 2 2025, 1:54 PM

tappof edited parent tasks, added: T321808: Port all Icinga checks to Prometheus/Alertmanager; removed: T333638: Icinga deprecation and sunsetting roadmap.Sep 2 2025, 4:33 PM

Change #1184715 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] wmcs: remove HighIOWaitStalling

https://gerrit.wikimedia.org/r/1184715

gerritbot added a project: Patch-For-Review.Sep 4 2025, 9:10 AM

Change #1184715 merged by Filippo Giunchedi:

[operations/alerts@master] wmcs: remove HighIOWaitStalling

https://gerrit.wikimedia.org/r/1184715

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2025, 9:32 AM

fnegri moved this task from Inbox to Epics on the cloud-services-team board.Sep 5 2025, 8:41 AM

Change #1200012 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] haproxy: enable nrpe2nodexp wrapper on check-cinder-snapshot-leaks

https://gerrit.wikimedia.org/r/1200012

gerritbot added a project: Patch-For-Review.Oct 30 2025, 9:48 AM

tappof mentioned this in T384472: Candidate nrpe checks for compatibility layer icinga/prometheus/alertmanager.Oct 30 2025, 10:16 AM

Change #1200016 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] neutron: enable nrpe2nodexp wrapper on check-neutron-conntrack

https://gerrit.wikimedia.org/r/1200016

Change #1200018 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] nova: enable nrpe2nodexp wrapper on check-flavor_aggregates

https://gerrit.wikimedia.org/r/1200018

Change #1200012 merged by Tiziano Fogli:

[operations/puppet@production] cinder: enable nrpe2nodexp wrapper on check-cinder-snapshot-leaks

https://gerrit.wikimedia.org/r/1200012

Change #1200016 merged by Tiziano Fogli:

[operations/puppet@production] neutron: enable nrpe2nodexp wrapper on check-neutron-conntrack

https://gerrit.wikimedia.org/r/1200016

Change #1200018 merged by Tiziano Fogli:

[operations/puppet@production] nova: enable nrpe2nodexp wrapper on check-flavor_aggregates

https://gerrit.wikimedia.org/r/1200018

Maintenance_bot removed a project: Patch-For-Review.Nov 12 2025, 1:30 PM

fgiunchedi edited projects, added: cloud-services-team (FY2025/2026-Q3-Q4); removed: cloud-services-team.Jan 7 2026, 10:27 AM

fgiunchedi updated the task description. (Show Details)Jun 9 2026, 7:33 AM

fgiunchedi updated the task description. (Show Details)Jun 9 2026, 7:58 AM

fgiunchedi closed subtask T345294: Move Cloud VPS control plane alerting to alertmanager as Resolved.Jun 9 2026, 8:08 AM

fgiunchedi removed a subtask: T347148: Determine how to monitor services in cloud-private / cloudlb.Jun 10 2026, 8:44 AM

fgiunchedi mentioned this in T345983: Remove Icinga checks for Cloud VPS projects (not: infrastructure).Jun 10 2026, 9:28 AM

fgiunchedi closed subtask T345983: Remove Icinga checks for Cloud VPS projects (not: infrastructure) as Resolved.Jun 15 2026, 10:20 AM

Change #1302114 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] openstack: deprecate ensure_running_kvm_instances check

https://gerrit.wikimedia.org/r/1302114

gerritbot added a project: Patch-For-Review.Jun 15 2026, 10:24 AM

Change #1302150 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] wmcs: do not pint-warn on NeutronAgentAdminDown

https://gerrit.wikimedia.org/r/1302150

Change #1302151 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] team-wmcs: introduce per-namespace neutron conntrack alert

https://gerrit.wikimedia.org/r/1302151

Change #1302164 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: remove 'key' label from neutron_netns metrics

https://gerrit.wikimedia.org/r/1302164

check-flavor_aggregates ?

It's useful to prevent creation of broken flavors (e.g. ones that don't specify between a ceph or localhost backend, or flavors without any IO throttling). That said, as long as flavors are only managed by cloud-vps admins it doesn't need to be checked this way. The future is probably having all flavors controlled by tofu-infra, in which case we can address the 'rogue flavor' issue by linting the tofu code.

ensure_running_kvm_instances should be removed in favor of higher level service (e.g. nova) signals

It's possible for nova services to be up and running on a host but all the actual VMs crashing (or simply not started after a reboot). I nevertheless think it's probably ok to try ripping out this particular check though, because I've seen lots of false positives and can't remember a time in the last several years where this was actually useful.

check-cinder-snapshot-leaks ?

This is watching for messes left by the backup system. It hasn't fired in ages and, in any case, if/when we revamp the ceph backup system this won't be meaningful.

Change #1302164 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: remove 'key' label from neutron_netns metrics

https://gerrit.wikimedia.org/r/1302164

Change #1302114 merged by Filippo Giunchedi:

[operations/puppet@production] openstack: deprecate ensure_running_kvm_instances check

https://gerrit.wikimedia.org/r/1302114

Change #1302748 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] openstack: deprecate icinga check-flavor_aggregates

https://gerrit.wikimedia.org/r/1302748

Change #1302749 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] openstack: deprecate check-cinder-snapshot-leaks

https://gerrit.wikimedia.org/r/1302749

Thank you for the comments / feedback !

In T328502#12020109, @Andrew wrote:

check-flavor_aggregates ?

It's useful to prevent creation of broken flavors (e.g. ones that don't specify between a ceph or localhost backend, or flavors without any IO throttling). That said, as long as flavors are only managed by cloud-vps admins it doesn't need to be checked this way. The future is probably having all flavors controlled by tofu-infra, in which case we can address the 'rogue flavor' issue by linting the tofu code.

Agreed, please take a look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302748

ensure_running_kvm_instances should be removed in favor of higher level service (e.g. nova) signals

It's possible for nova services to be up and running on a host but all the actual VMs crashing (or simply not started after a reboot). I nevertheless think it's probably ok to try ripping out this particular check though, because I've seen lots of false positives and can't remember a time in the last several years where this was actually useful.

Thank you, {{done}} in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302114

check-cinder-snapshot-leaks ?

This is watching for messes left by the backup system. It hasn't fired in ages and, in any case, if/when we revamp the ceph backup system this won't be meaningful.

Makes sense, I have this review out https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302749

Change #1302764 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] openstack: deprecate icinga check-neutron-conntrack

https://gerrit.wikimedia.org/r/1302764

Change #1302150 merged by Filippo Giunchedi:

[operations/alerts@master] team-wmcs: do not pint-warn on NeutronAgentAdminDown

https://gerrit.wikimedia.org/r/1302150

Change #1302749 merged by Filippo Giunchedi:

[operations/puppet@production] openstack: deprecate check-cinder-snapshot-leaks

https://gerrit.wikimedia.org/r/1302749

fgiunchedi created subtask T429451: Establish a blackbox network probe vantage point into cloud realm.Jun 17 2026, 9:11 AM

Change #1302748 merged by Filippo Giunchedi:

[operations/puppet@production] openstack: deprecate icinga check-flavor_aggregates

https://gerrit.wikimedia.org/r/1302748

fgiunchedi updated the task description. (Show Details)Jun 18 2026, 1:02 PM

Change #1302151 merged by Filippo Giunchedi:

[operations/alerts@master] team-wmcs: introduce per-namespace neutron conntrack alert

https://gerrit.wikimedia.org/r/1302151

Change #1307762 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/alerts@master] team-wmcs: Port Memcached availability alert

https://gerrit.wikimedia.org/r/1307762

Change #1307764 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Disable Memcached Icinga checks for cloudcontrols/cloudwebs

https://gerrit.wikimedia.org/r/1307764

Change #1307762 merged by jenkins-bot:

[operations/alerts@master] team-wmcs: Port Memcached availability alert

https://gerrit.wikimedia.org/r/1307762

Change #1307764 merged by Majavah:

[operations/puppet@production] Disable Memcached Icinga checks for cloudcontrols/cloudwebs

https://gerrit.wikimedia.org/r/1307764

Change #1308626 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] openstack: remove openstack::monitor::neutron::l3_agent_conntrack

https://gerrit.wikimedia.org/r/1308626

Change #1302764 merged by Filippo Giunchedi:

[operations/puppet@production] openstack: deprecate icinga check-neutron-conntrack

https://gerrit.wikimedia.org/r/1302764

Change #1308626 merged by Filippo Giunchedi:

[operations/puppet@production] openstack: remove openstack::monitor::neutron::l3_agent_conntrack

https://gerrit.wikimedia.org/r/1308626

Herald added projects: tools-infrastructure-team, tools-platform-team. · View Herald TranscriptJul 13 2026, 8:18 AM

Maintenance_bot removed a project: Patch-For-Review.Jul 13 2026, 8:31 AM

fnegri removed projects: cloud-services-team (FY2025/2026-Q3-Q4), Goal.Jul 13 2026, 3:14 PM

Change #1312960 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] team-wmcs: add wikitech alerts

https://gerrit.wikimedia.org/r/1312960

gerritbot added a project: Patch-For-Review.Jul 21 2026, 6:41 AM

fgiunchedi closed subtask T362397: Move wikitech-static monitoring off Icinga as Resolved.Aug 19 2026, 1:05 PM

Ok we're very close to the finish line! What's left are the DNS checks from production, which hinge on a blackbox-exporter instance being able to reach cloud and being query-able from prod (i.e. T429451)

To be honest since we're already checking dns from metricsinfra, I'm tempted to just go ahead and remove those checks from icinga too. If DNS is badly broken other stuff will break anyways and monitoring will complain, let me know what you think.

T429451: Establish a blackbox network probe vantage point into cloud realm is still worthwhile doing/having on its own, and I'll leave that task open

In T328502#12231204, @fgiunchedi wrote:

To be honest since we're already checking dns from metricsinfra, I'm tempted to just go ahead and remove those checks from icinga too. If DNS is badly broken other stuff will break anyways and monitoring will complain, let me know what you think.

Does that mean we would fail to alert in a scenario where DNS works perfectly inside the cloud but no one in the outside world can resolve e.g. paws.wmcloud.org? That seems bad... certainly other things would probably alert but I can imagine convincing myself that it's not DNS (even though it's always DNS).

In T328502#12237329, @Andrew wrote:

In T328502#12231204, @fgiunchedi wrote:

To be honest since we're already checking dns from metricsinfra, I'm tempted to just go ahead and remove those checks from icinga too. If DNS is badly broken other stuff will break anyways and monitoring will complain, let me know what you think.

Does that mean we would fail to alert in a scenario where DNS works perfectly inside the cloud but no one in the outside world can resolve e.g. paws.wmcloud.org? That seems bad... certainly other things would probably alert but I can imagine convincing myself that it's not DNS (even though it's always DNS).

Yes in said scenario other things would alert, including end to end tests, hence my reasoning of leaving dns tests within metricsinfra and complete the icinga deprecation for wmcs

In T328502#12261221, @fgiunchedi wrote:

In T328502#12237329, @Andrew wrote:

In T328502#12231204, @fgiunchedi wrote:

To be honest since we're already checking dns from metricsinfra, I'm tempted to just go ahead and remove those checks from icinga too. If DNS is badly broken other stuff will break anyways and monitoring will complain, let me know what you think.

Does that mean we would fail to alert in a scenario where DNS works perfectly inside the cloud but no one in the outside world can resolve e.g. paws.wmcloud.org? That seems bad... certainly other things would probably alert but I can imagine convincing myself that it's not DNS (even though it's always DNS).

Yes in said scenario other things would alert, including end to end tests, hence my reasoning of leaving dns tests within metricsinfra and complete the icinga deprecation for wmcs

Ok! Worth it to get rid of icinga!

andrea.denisse unsubscribed.Aug 27 2026, 3:30 PM

Change #1332750 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] openstack: stop using icinga for pdns auth checks

https://gerrit.wikimedia.org/r/1332750

Change #1332751 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] openldap: stop using icinga in clouddev

https://gerrit.wikimedia.org/r/1332751

Change #1332750 merged by Filippo Giunchedi: