Page MenuHomePhabricator
Create Task
Maniphest T384472

Candidate nrpe checks for compatibility layer icinga/prometheus/alertmanager
Open, MediumPublicGoal
Actions

Description

Candidates

The puppet snippets below are an audit (as of commit 36a2a9ea99e0ff) of nrpe checks using scripts without an immediate prometheus-ready replacement. Note that the usual disclaimer "some of these might be not relevant anymore" applies.

check_microcode

( T350694: Infrastructure Foundation Alerts to migrate - deprecated - patch submitted)

modules/base/manifests/kernel.pp:    nrpe::monitor_service { 'cpu_microcode_status':
modules/base/manifests/kernel.pp-        ensure         => 'present',
modules/base/manifests/kernel.pp-        description    => 'Check whether microcode mitigations for CPU vulnerabilities are applied',
modules/base/manifests/kernel.pp-        nrpe_command   => '/usr/local/lib/nagios/plugins/check_microcode',
modules/base/manifests/kernel.pp-        contact_group  => 'admins',
modules/base/manifests/kernel.pp-        check_interval => 1440,
modules/base/manifests/kernel.pp-        retry_interval => 5,
modules/base/manifests/kernel.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/Microcode',
modules/base/manifests/kernel.pp-    }
modules/base/manifests/kernel.pp-
check_gdnsd_checkconf

(T384425: Port DNS icinga checks to Alertmanager - icinga: enabled - nrpe2nodexp: testing - final recipient: sre)

modules/gdnsd/manifests/monitor_conf.pp:    nrpe::monitor_service { 'gdnsd_checkconf':
modules/gdnsd/manifests/monitor_conf.pp-        description  => 'gdnsd checkconf',
modules/gdnsd/manifests/monitor_conf.pp-        nrpe_command => '/usr/local/lib/nagios/plugins/check_gdnsd_checkconf',
modules/gdnsd/manifests/monitor_conf.pp-        critical     => true,
modules/gdnsd/manifests/monitor_conf.pp-        notes_url    => 'https://wikitech.wikimedia.org/wiki/DNS#gdnsd_checkconf',
modules/gdnsd/manifests/monitor_conf.pp-    }
check_haproxy

(T407137: Port haproxy checks to prometheus/alertmanager - icinga: enabled - nrpe2nodexp: testing - final recipient: sre)

modules/haproxy/manifests/init.pp:        nrpe::monitor_service { 'haproxy_alive':
modules/haproxy/manifests/init.pp-            ensure       => bool2str($monitor_check_haproxy, 'present', 'absent'),
modules/haproxy/manifests/init.pp-            description  => 'haproxy alive',
modules/haproxy/manifests/init.pp-            nrpe_command => '/usr/local/lib/nagios/plugins/check_haproxy --check=alive',
modules/haproxy/manifests/init.pp-            notes_url    => 'https://wikitech.wikimedia.org/wiki/HAProxy',
modules/haproxy/manifests/init.pp-        }
modules/profile/manifests/mariadb/proxy/master.pp:    nrpe::monitor_service { 'haproxy_failover':
modules/profile/manifests/mariadb/proxy/master.pp-        description  => 'haproxy failover',
modules/profile/manifests/mariadb/proxy/master.pp-        nrpe_command => '/usr/local/lib/nagios/plugins/check_haproxy --check=failover',
modules/profile/manifests/mariadb/proxy/master.pp-        notes_url    => 'https://wikitech.wikimedia.org/wiki/HAProxy',
modules/profile/manifests/mariadb/proxy/master.pp-    }
check_cirrus_settings.py
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp:    nrpe::monitor_service { "elasticsearch_setting_check_${port}":
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp-        critical       => false,
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp-        contact_group  => 'admins,team-discovery',
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/Search#Administration',
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp-        nrpe_command   => "/usr/lib/nagios/plugins/check_cirrus_settings.py --url http://localhost:${port}  --settings-file /etc/elasticsearch/${title}/cirrus_check_settings.yaml",
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp-        description    => "ElasticSearch setting check - ${port}",
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp-        check_interval => 720, # 12h
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp-        retry_interval => 120, # 2h
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp-        retries        => 1,
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp-    }
check-mariadb-backups

(T315866: Migrate mysql icinga alerts to alert manager - icinga: enabled, nrpe2nodexp: testing, final recipient: data persistence)

modules/dbbackups/manifests/check.pp:    nrpe::monitor_service { "mariadb_${type}_${section}_${datacenter}":
modules/dbbackups/manifests/check.pp-        description    => "${type} of ${section} in ${datacenter}",
modules/dbbackups/manifests/check.pp-        nrpe_command   => $check_command,
modules/dbbackups/manifests/check.pp-        critical       => false,
modules/dbbackups/manifests/check.pp-        contact_group  => 'admins',
modules/dbbackups/manifests/check.pp-        sudo_user      => 'backupcheck',
modules/dbbackups/manifests/check.pp-        check_interval => 30,  # Don't check too often
modules/dbbackups/manifests/check.pp-        require        => [
modules/dbbackups/manifests/check.pp-            Package['wmfbackups-check'],
modules/dbbackups/manifests/check.pp-            File['/etc/wmfbackups/valid_sections.txt'],
modules/dbbackups/manifests/check.pp-            User['backupcheck'],
modules/dbbackups/manifests/check.pp-        ],
modules/dbbackups/manifests/check.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Rerun_a_failed_backup',
modules/dbbackups/manifests/check.pp-    }
modules/dbbackups/manifests/check.pp-}
db-check-health
modules/mariadb/manifests/monitor_eventscheduler.pp:    nrpe::monitor_service { "mariadb_event_scheduler_${name}":
modules/mariadb/manifests/monitor_eventscheduler.pp-        description   => "MariaDB Event Scheduler ${name}",
modules/mariadb/manifests/monitor_eventscheduler.pp-        nrpe_command  => $check_command,
modules/mariadb/manifests/monitor_eventscheduler.pp-        critical      => $is_critical,
modules/mariadb/manifests/monitor_eventscheduler.pp-        contact_group => $contact_group,
modules/mariadb/manifests/monitor_eventscheduler.pp-        require       => Package['wmfmariadbpy-common'],
modules/mariadb/manifests/monitor_eventscheduler.pp-        notes_url     => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Event_Scheduler',
modules/mariadb/manifests/monitor_eventscheduler.pp-    }
modules/mariadb/manifests/monitor_eventscheduler.pp-}
modules/mariadb/manifests/monitor_readonly.pp:    nrpe::monitor_service { "mariadb_read_only_${name}":
modules/mariadb/manifests/monitor_readonly.pp-        description   => "MariaDB read only ${name}",
modules/mariadb/manifests/monitor_readonly.pp-        nrpe_command  => $check_command,
modules/mariadb/manifests/monitor_readonly.pp-        critical      => $is_critical,
modules/mariadb/manifests/monitor_readonly.pp-        contact_group => $contact_group,
modules/mariadb/manifests/monitor_readonly.pp-        require       => Package['wmfmariadbpy-common'],
modules/mariadb/manifests/monitor_readonly.pp-        notes_url     => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Master_comes_back_in_read_only',
modules/mariadb/manifests/monitor_readonly.pp-    }
modules/mariadb/manifests/monitor_readonly.pp-}
pmp-check-unix-memory
modules/mariadb/manifests/monitor_memory.pp:    nrpe::monitor_service { 'mariadb_memory':
modules/mariadb/manifests/monitor_memory.pp-        description   => 'MariaDB memory',
modules/mariadb/manifests/monitor_memory.pp-        nrpe_command  => $check_command,
modules/mariadb/manifests/monitor_memory.pp-        critical      => $is_critical,
modules/mariadb/manifests/monitor_memory.pp-        contact_group => $contact_group,
modules/mariadb/manifests/monitor_memory.pp-        require       => Package['monitoring-plugins-contrib'],
modules/mariadb/manifests/monitor_memory.pp-        notes_url     => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting',
modules/mariadb/manifests/monitor_memory.pp-    }
modules/mariadb/manifests/monitor_memory.pp-}
check_mariadb
modules/mariadb/manifests/monitor_replication.pp:    nrpe::monitor_service { "mariadb_replica_io_state_${name}":
modules/mariadb/manifests/monitor_replication.pp-        description   => "MariaDB Replica IO: ${name}",
modules/mariadb/manifests/monitor_replication.pp-        nrpe_command  => "${check_mariadb} --check=slave_io_state",
modules/mariadb/manifests/monitor_replication.pp-        critical      => $is_critical,
modules/mariadb/manifests/monitor_replication.pp-        contact_group => $contact_group,
modules/mariadb/manifests/monitor_replication.pp-        notes_url     => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica',
modules/mariadb/manifests/monitor_replication.pp-    }
modules/mariadb/manifests/monitor_replication.pp-
modules/mariadb/manifests/monitor_replication.pp:    nrpe::monitor_service { "mariadb_replica_sql_state_${name}":
modules/mariadb/manifests/monitor_replication.pp-        description   => "MariaDB Replica SQL: ${name}",
modules/mariadb/manifests/monitor_replication.pp-        nrpe_command  => "${check_mariadb} --check=slave_sql_state",
modules/mariadb/manifests/monitor_replication.pp-        critical      => $is_critical,
modules/mariadb/manifests/monitor_replication.pp-        contact_group => $contact_group,
modules/mariadb/manifests/monitor_replication.pp-        notes_url     => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica',
modules/mariadb/manifests/monitor_replication.pp-    }
modules/mariadb/manifests/monitor_replication.pp-
modules/mariadb/manifests/monitor_replication.pp-    # check the lag towards the $source_dc's master
modules/mariadb/manifests/monitor_replication.pp:    nrpe::monitor_service { "mariadb_replica_sql_lag_${name}":
modules/mariadb/manifests/monitor_replication.pp-        description   => "MariaDB Replica Lag: ${name}",
modules/mariadb/manifests/monitor_replication.pp-        nrpe_command  => "${check_mariadb} --check=slave_sql_lag \
modules/mariadb/manifests/monitor_replication.pp-                          --shard=${name} --datacenter=${source_dc} \
modules/mariadb/manifests/monitor_replication.pp-                          --sql-lag-warn=${lag_warn} \
modules/mariadb/manifests/monitor_replication.pp-                          --sql-lag-crit=${lag_crit}",
modules/mariadb/manifests/monitor_replication.pp-        retries       => 10,
modules/mariadb/manifests/monitor_replication.pp-        critical      => $is_critical,
modules/mariadb/manifests/monitor_replication.pp-        contact_group => $contact_group,
modules/mariadb/manifests/monitor_replication.pp-        notes_url     => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica',
modules/mariadb/manifests/monitor_replication.pp-    }
modules/mariadb/manifests/monitor_replication.pp-}
check_dir-not-bad-owner

(T407120: O11y alerts to migrate - icinga: enabled, nrpe2nodex: testing, final recipient: sre)

modules/monitoring/manifests/icinga/bad_directory_owner.pp:    nrpe::monitor_service { "${safe_title}_owned":
modules/monitoring/manifests/icinga/bad_directory_owner.pp-        description    => "Improperly owned (${uid}:${gid}) files in ${title}",
modules/monitoring/manifests/icinga/bad_directory_owner.pp-        nrpe_command   => $filename,
modules/monitoring/manifests/icinga/bad_directory_owner.pp-        check_interval => $interval,
modules/monitoring/manifests/icinga/bad_directory_owner.pp-        timeout        => $timeout,
modules/monitoring/manifests/icinga/bad_directory_owner.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/Monitoring/bad_directory_owner',
modules/monitoring/manifests/icinga/bad_directory_owner.pp-    }
modules/monitoring/manifests/icinga/bad_directory_owner.pp-}
check_cinder_snapshot_leaks.py

(T328502: Move WMCS off of Icinga and introduce alertmanager - icinga: enabled, nrpe2nodex: testing, final recipient: wmcs only)

modules/openstack/manifests/cinder/monitor.pp:    nrpe::monitor_service { 'check-cinder-snapshot-leaks':
modules/openstack/manifests/cinder/monitor.pp-        ensure        => 'present',
modules/openstack/manifests/cinder/monitor.pp-        nrpe_command  => '/usr/local/bin/check_cinder_snapshot_leaks.py',
modules/openstack/manifests/cinder/monitor.pp-        description   => 'Check for snapshots leaked by cinder backup agent',
modules/openstack/manifests/cinder/monitor.pp-        require       => File['/usr/local/bin/check_cinder_snapshot_leaks.py'],
modules/openstack/manifests/cinder/monitor.pp-        contact_group => 'wmcs-team-email,admins',
modules/openstack/manifests/cinder/monitor.pp-        notes_url     => 'https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_for_snapshots_leaked_by_cinder_backup_agent',
modules/openstack/manifests/cinder/monitor.pp-    }
check_neutron_conntrack

(T328502: Move WMCS off of Icinga and introduce alertmanager - icinga: enabled, nrpe2nodex: patch submitted, final recipient: wmcs)

modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp:    nrpe::monitor_service { 'check-neutron-conntrack':
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp-        ensure        => 'present',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp-        nrpe_command  => '/usr/local/lib/nagios/plugins/check_neutron_conntrack',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp-        sudo_user     => 'root',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp-        description   => 'Check nf_conntrack usage in neutron netns',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp-        contact_group => 'wmcs-team-email,admins',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp-        notes_url     => 'https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp-    }
check_flavor_properties

(T328502: Move WMCS off of Icinga and introduce alertmanager - icinga: enabled, nrpe2nodex: patch submitted, final recipient: wmcs)

modules/openstack/manifests/nova/fullstack/monitor.pp:    nrpe::monitor_service { 'check-flavor_aggregates':
modules/openstack/manifests/nova/fullstack/monitor.pp-        ensure         => 'present',
modules/openstack/manifests/nova/fullstack/monitor.pp-        nrpe_command   => '/usr/local/lib/nagios/plugins/check_flavor_properties',
modules/openstack/manifests/nova/fullstack/monitor.pp-        sudo_user      => 'root',
modules/openstack/manifests/nova/fullstack/monitor.pp-        description    => 'all nova flavors are assigned necessary properties',
modules/openstack/manifests/nova/fullstack/monitor.pp-        timeout        => 30,
modules/openstack/manifests/nova/fullstack/monitor.pp-        check_interval => 15,
modules/openstack/manifests/nova/fullstack/monitor.pp-        contact_group  => 'wmcs-team-email,wmcs-bots',
modules/openstack/manifests/nova/fullstack/monitor.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Host_aggregates';
modules/openstack/manifests/nova/fullstack/monitor.pp-    }
check_dbctl_uncommitted_diffs

(T350694: Infrastructure Foundation Alerts to migrate - icinga: enabled, nrpe2nodex: patch submitted, final recipient: ???)

modules/profile/manifests/conftool/dbctl_client.pp:    nrpe::monitor_service { 'dbctl_uncommitted_diffs':
modules/profile/manifests/conftool/dbctl_client.pp-        ensure         => present,
modules/profile/manifests/conftool/dbctl_client.pp-        description    => 'Uncommitted dbctl configuration changes, check dbctl config diff',
modules/profile/manifests/conftool/dbctl_client.pp-        nrpe_command   => '/usr/local/lib/nagios/plugins/check_dbctl_uncommitted_diffs',
modules/profile/manifests/conftool/dbctl_client.pp-        critical       => false,
modules/profile/manifests/conftool/dbctl_client.pp-        check_interval => 5,
modules/profile/manifests/conftool/dbctl_client.pp-        retry_interval => 5,
modules/profile/manifests/conftool/dbctl_client.pp-        retries        => 3,
modules/profile/manifests/conftool/dbctl_client.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/Dbctl#Uncommitted_dbctl_diffs',
modules/profile/manifests/conftool/dbctl_client.pp-        timeout        => 20,
modules/profile/manifests/conftool/dbctl_client.pp-    }
check_dotls

(T384425: Port DNS icinga checks to Alertmanager - icinga: enabled, nrpe2nodex: patch submitted, final recipient: ???)

modules/profile/manifests/dns/auth/dotls.pp:    nrpe::monitor_service { 'check_dotls':
modules/profile/manifests/dns/auth/dotls.pp-        description  => 'AuthDNS-over-TLS Works',
modules/profile/manifests/dns/auth/dotls.pp-        nrpe_command => '/usr/local/lib/nagios/plugins/check_dotls',
modules/profile/manifests/dns/auth/dotls.pp-        notes_url    => 'https://wikitech.wikimedia.org/wiki/DNS',
modules/profile/manifests/dns/auth/dotls.pp-    }
check_authdns_update_run

(T384425: Port DNS icinga checks to Alertmanager - icinga: enabled, nrpe2nodex: patch submitted, final recipient: sre)

modules/profile/manifests/dns/auth/update.pp:    nrpe::monitor_service { 'authdns_update_run':
modules/profile/manifests/dns/auth/update.pp-        description    => 'check if authdns-update was run after a change was merged to operations/dns.git',
modules/profile/manifests/dns/auth/update.pp-        nrpe_command   => '/usr/local/lib/nagios/plugins/check_authdns_update_run',
modules/profile/manifests/dns/auth/update.pp-        check_interval => 5, # min
modules/profile/manifests/dns/auth/update.pp-        retry_interval => 5, # min
modules/profile/manifests/dns/auth/update.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/DNS#authdns_update_run',
modules/profile/manifests/dns/auth/update.pp-    }
check_ferm

(T350694: Infrastructure Foundation Alerts to migrate - icinga: enabled, nrpe2nodexp: testing, final recipient: sre)

modules/profile/manifests/firewall.pp:            nrpe::monitor_service { 'ferm_active':
modules/profile/manifests/firewall.pp-                description    => 'Check whether ferm is active by checking the default input chain',
modules/profile/manifests/firewall.pp-                nrpe_command   => '/usr/local/lib/nagios/plugins/check_ferm',
modules/profile/manifests/firewall.pp-                sudo_user      => 'root',
modules/profile/manifests/firewall.pp-                contact_group  => 'admins',
modules/profile/manifests/firewall.pp-                notes_url      => 'https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm',
modules/profile/manifests/firewall.pp-                check_interval => 30,
modules/profile/manifests/firewall.pp-                retry_interval => $ferm_icinga_retry_interval,
modules/profile/manifests/firewall.pp-            }
check_sysctl

(T407330: Port lvs checks to Prometheus/Alertmanager - icinga: enabled, nrpe2nodex: patch submitted, final recipient: ???)

modules/profile/manifests/lvs.pp:    nrpe::monitor_service { 'check_rp_filter_disabled':
modules/profile/manifests/lvs.pp-        description  => 'Check rp_filter disabled',
modules/profile/manifests/lvs.pp-        nrpe_command => "/usr/local/lib/nagios/plugins/check_sysctl ${rp_args}",
modules/profile/manifests/lvs.pp-        notes_url    => 'https://wikitech.wikimedia.org/wiki/Monitoring/check_rp_filter_disabled',
modules/profile/manifests/lvs.pp-    }
check_exim_queue

(deprecated)

modules/profile/manifests/mail/smarthost.pp:    nrpe::monitor_service { 'check_exim_queue':
modules/profile/manifests/mail/smarthost.pp-        description    => 'exim queue',
modules/profile/manifests/mail/smarthost.pp-        nrpe_command   => '/usr/local/lib/nagios/plugins/check_exim_queue -w 1000 -c 3000',
modules/profile/manifests/mail/smarthost.pp-        check_interval => 30,
modules/profile/manifests/mail/smarthost.pp-        retry_interval => 10,
modules/profile/manifests/mail/smarthost.pp-        timeout        => 20,
modules/profile/manifests/mail/smarthost.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/Mail#Troubleshooting',
modules/profile/manifests/mail/smarthost.pp-    }
check_json_file

(T350694: Infrastructure Foundation Alerts to migrate - icinga: enabled, nrpe2nodexp: patch submitted, final recipient: dcops)

modules/profile/manifests/netbox/automation.pp:    nrpe::monitor_service { 'check_uncommitted_dns_changes':
modules/profile/manifests/netbox/automation.pp-        ensure         => $active_ensure,
modules/profile/manifests/netbox/automation.pp-        check_interval => 5,
modules/profile/manifests/netbox/automation.pp-        retry_interval => 2,
modules/profile/manifests/netbox/automation.pp-        description    => 'Uncommitted DNS changes in Netbox',
modules/profile/manifests/netbox/automation.pp-        nrpe_command   => "${check_command} ${icinga_state_file} ${max_age}",
modules/profile/manifests/netbox/automation.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/Monitoring/Netbox_DNS_uncommitted_changes',
modules/profile/manifests/netbox/automation.pp-        contact_group  => 'team-dcops',
modules/profile/manifests/netbox/automation.pp-    }
check_service_restart
modules/profile/manifests/ntp.pp:        nrpe::monitor_service { "check_service_restart_${service}":
modules/profile/manifests/ntp.pp-            description    => "Check if ${service} has been restarted after ${conf_file} was changed",
modules/profile/manifests/ntp.pp-            nrpe_command   => "/usr/local/lib/nagios/plugins/check_ntp_service --service ${service} --file ${conf_file} --critical 2",
modules/profile/manifests/ntp.pp-            sudo_user      => 'root',
modules/profile/manifests/ntp.pp-            check_interval => 60, # 60mins
modules/profile/manifests/ntp.pp-            retry_interval => 30, # 30mins
modules/profile/manifests/ntp.pp-            notes_url      => 'https://wikitech.wikimedia.org/wiki/NTP#Monitoring',
modules/profile/manifests/ntp.pp-        }
modules/profile/manifests/pybal.pp:    nrpe::monitor_service { 'check_service_restart_pybal':
modules/profile/manifests/pybal.pp-        description    => 'Check if Pybal has been restarted after pybal.conf was changed',
modules/profile/manifests/pybal.pp-        nrpe_command   => '/usr/local/lib/nagios/plugins/check_pybal_restart --service pybal.service --file /etc/pybal/pybal.conf',
modules/profile/manifests/pybal.pp-        check_interval => 120, # 120mins
modules/profile/manifests/pybal.pp-        retry_interval => 60,  # 60mins
modules/profile/manifests/pybal.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/PyBal#Pybal_service_has_not_been_restarted',
modules/profile/manifests/pybal.pp-    }
modules/profile/manifests/wikidough.pp:        nrpe::monitor_service { "check_service_restart_${service}":
modules/profile/manifests/wikidough.pp-            description    => "Check if ${service} has been restarted after ${conf_file} was changed",
modules/profile/manifests/wikidough.pp-            nrpe_command   => "/usr/local/lib/nagios/plugins/check_wikidough_restart --service ${service} --file ${conf_file}",
modules/profile/manifests/wikidough.pp-            sudo_user      => 'root',
modules/profile/manifests/wikidough.pp-            check_interval => 360,  # 6h
modules/profile/manifests/wikidough.pp-            retry_interval => 60,   # 1h
modules/profile/manifests/wikidough.pp-            notes_url      => 'https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring#Service_Restart_Check',
modules/profile/manifests/wikidough.pp-        }
check_timedatectl
modules/profile/manifests/systemd/timesyncd.pp:    nrpe::monitor_service { 'timesynd_ntp_status':
modules/profile/manifests/systemd/timesyncd.pp-        ensure         => $ensure,
modules/profile/manifests/systemd/timesyncd.pp-        description    => 'Check the NTP synchronisation status of timesyncd',
modules/profile/manifests/systemd/timesyncd.pp-        nrpe_command   => '/usr/local/lib/nagios/plugins/check_timedatectl',
modules/profile/manifests/systemd/timesyncd.pp-        contact_group  => 'admins',
modules/profile/manifests/systemd/timesyncd.pp-        check_interval => 30,
modules/profile/manifests/systemd/timesyncd.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/NTP',
modules/profile/manifests/systemd/timesyncd.pp-    }
check_default_ats_lua_conf
modules/profile/manifests/trafficserver/backend.pp:        nrpe::monitor_service { 'default_ats_lua_conf':
modules/profile/manifests/trafficserver/backend.pp-            description  => 'Default ATS Lua configuration file',
modules/profile/manifests/trafficserver/backend.pp-            nrpe_command => '/usr/local/lib/nagios/plugins/check_default_ats_lua_conf',
modules/profile/manifests/trafficserver/backend.pp-            notes_url    => 'https://wikitech.wikimedia.org/wiki/ATS',
modules/profile/manifests/trafficserver/backend.pp-        }
check_journal_pattern
modules/profile/manifests/trafficserver/logs.pp:    nrpe::monitor_service { "${service_name}_skipped_logs":
modules/profile/manifests/trafficserver/logs.pp-        description  => "Logs skipped by ${service_name}",
modules/profile/manifests/trafficserver/logs.pp-        nrpe_command => "/usr/local/lib/nagios/plugins/check_journal_pattern '1 hour ago' 'NOTE: Skipping the current log entry for ' ${service_name}",
modules/profile/manifests/trafficserver/logs.pp-        notes_url    => 'https://wikitech.wikimedia.org/wiki/ATS',
modules/profile/manifests/trafficserver/logs.pp-    }
check_pybal
modules/pybal/manifests/monitoring.pp:    nrpe::monitor_service { 'pybal_backends':
modules/pybal/manifests/monitoring.pp-        description  => 'PyBal backends health check',
modules/pybal/manifests/monitoring.pp-        nrpe_command => '/usr/local/lib/nagios/plugins/check_pybal --url http://localhost:9090/alerts',
modules/pybal/manifests/monitoring.pp-        notes_url    => 'https://wikitech.wikimedia.org/wiki/PyBal',
modules/pybal/manifests/monitoring.pp-    }
check_pybal_ipvs_diff
modules/pybal/manifests/monitoring.pp:    nrpe::monitor_service { 'pybal_ipvs_diff':
modules/pybal/manifests/monitoring.pp-        description    => 'PyBal IPVS diff check',
modules/pybal/manifests/monitoring.pp-        nrpe_command   => "/usr/local/lib/nagios/plugins/check_pybal_ipvs_diff --req-timeout=10.0 --prometheus-url http://${::ipaddress}:9100/metrics",
modules/pybal/manifests/monitoring.pp-        check_interval => 5,
modules/pybal/manifests/monitoring.pp-        timeout        => 60,
modules/pybal/manifests/monitoring.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/PyBal',
modules/pybal/manifests/monitoring.pp-    }
check_established_connections
modules/pybal/manifests/monitoring.pp:        nrpe::monitor_service { 'pybal_etcd_connections':
modules/pybal/manifests/monitoring.pp-            description    => 'PyBal connections to etcd',
modules/pybal/manifests/monitoring.pp-            nrpe_command   => "/usr/local/lib/nagios/plugins/check_established_connections ${config_host} ${etcd_port} ${n_etcd_connections}",
modules/pybal/manifests/monitoring.pp-            check_interval => 5,
modules/pybal/manifests/monitoring.pp-            timeout        => 60,
modules/pybal/manifests/monitoring.pp-            notes_url      => 'https://wikitech.wikimedia.org/wiki/PyBal',
modules/pybal/manifests/monitoring.pp-        }
check_hpssacli
modules/raid/manifests/hpsa/hpssacli.pp:  nrpe::monitor_service { 'raid_hpssacli':
modules/raid/manifests/hpsa/hpssacli.pp-    description    => 'HP RAID',
modules/raid/manifests/hpsa/hpssacli.pp-    nrpe_command   => '/usr/local/lib/nagios/plugins/check_hpssacli',
modules/raid/manifests/hpsa/hpssacli.pp-    timeout        => 90, # can take > 10s on servers with lots of disks
modules/raid/manifests/hpsa/hpssacli.pp-    check_interval => $raid::check_interval,
modules/raid/manifests/hpsa/hpssacli.pp-    retry_interval => $raid::retry_interval,
modules/raid/manifests/hpsa/hpssacli.pp-    event_handler  => "raid_handler!hpssacli!${::site}",
modules/raid/manifests/hpsa/hpssacli.pp-    notes_url      => 'https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook#Hardware_Raid_Information_Gathering',
modules/raid/manifests/hpsa/hpssacli.pp-  }
check_ssacli
modules/raid/manifests/hpsa/ssacli.pp:    nrpe::monitor_service { 'raid_ssacli':
modules/raid/manifests/hpsa/ssacli.pp-        description    => 'HP RAID',
modules/raid/manifests/hpsa/ssacli.pp-        nrpe_command   => '/usr/local/lib/nagios/plugins/check_ssacli',
modules/raid/manifests/hpsa/ssacli.pp-        timeout        => 90, # can take > 10s on servers with lots of disks
modules/raid/manifests/hpsa/ssacli.pp-        check_interval => $raid::check_interval,
modules/raid/manifests/hpsa/ssacli.pp-        retry_interval => $raid::retry_interval,
modules/raid/manifests/hpsa/ssacli.pp-        event_handler  => "raid_handler!ssacli!${::site}",
modules/raid/manifests/hpsa/ssacli.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook#Hardware_Raid_Information_Gathering',
modules/raid/manifests/hpsa/ssacli.pp-    }
check_megacli
modules/raid/manifests/megaraid.pp:  nrpe::monitor_service { 'raid_megaraid':
modules/raid/manifests/megaraid.pp-    description    => 'MegaRAID',
modules/raid/manifests/megaraid.pp-    nrpe_command   => "${raid::check_raid} megacli",
modules/raid/manifests/megaraid.pp-    sudo_user      => 'root',
modules/raid/manifests/megaraid.pp-    check_interval => $raid::check_interval,
modules/raid/manifests/megaraid.pp-    retry_interval => $raid::retry_interval,
modules/raid/manifests/megaraid.pp-    event_handler  => "raid_handler!megacli!${::site}",
modules/raid/manifests/megaraid.pp-    notes_url      => 'https://wikitech.wikimedia.org/wiki/MegaCli#Monitoring',
modules/raid/manifests/megaraid.pp-  }
check_perccli
modules/raid/manifests/perccli.pp:    nrpe::monitor_service { 'raid_perc_raid':
modules/raid/manifests/perccli.pp-        description    => 'Dell PowerEdge RAID Controller',
modules/raid/manifests/perccli.pp-        nrpe_command   => '/usr/local/lib/nagios/plugins/get-raid-status-perccli',
modules/raid/manifests/perccli.pp-        sudo_user      => 'root',
modules/raid/manifests/perccli.pp-        check_interval => $raid::check_interval,
modules/raid/manifests/perccli.pp-        retry_interval => $raid::retry_interval,
modules/raid/manifests/perccli.pp-        event_handler  => "raid_handler!perccli!${::site}",
modules/raid/manifests/perccli.pp-        notes_url      => 'https://wikitech.wikimedia.org/wiki/PERCCli#Monitoring',
modules/raid/manifests/perccli.pp-    }
check_udp2log_log_age
modules/udp2log/manifests/instance/monitoring.pp:        nrpe::monitor_service { "udp2log_log_age-${name}":
modules/udp2log/manifests/instance/monitoring.pp-            ensure        => 'present',
modules/udp2log/manifests/instance/monitoring.pp-            description   => "udp2log log age for ${name}",
modules/udp2log/manifests/instance/monitoring.pp-            nrpe_command  => "/usr/lib/nagios/plugins/check_udp2log_log_age ${name}",
modules/udp2log/manifests/instance/monitoring.pp-            contact_group => 'admins,analytics,team-data-platform',
modules/udp2log/manifests/instance/monitoring.pp-            notes_url     => 'https://wikitech.wikimedia.org/wiki/Udp2log',
modules/udp2log/manifests/instance/monitoring.pp-        }
check_vcl_reload
modules/varnish/manifests/common/director_scripts.pp:    nrpe::monitor_service { 'confd_vcl_reload':
modules/varnish/manifests/common/director_scripts.pp-        description  => 'Confd vcl based reload',
modules/varnish/manifests/common/director_scripts.pp-        nrpe_command => '/usr/local/lib/nagios/plugins/check_vcl_reload',
modules/varnish/manifests/common/director_scripts.pp-        notes_url    => 'https://wikitech.wikimedia.org/wiki/Varnish',
modules/varnish/manifests/common/director_scripts.pp-    }
check_varnish_uds
modules/varnish/manifests/instance.pp:            nrpe::monitor_service { "check-varnish-uds${instancesuffix}-${uds_path}":
modules/varnish/manifests/instance.pp-                ensure       => present,
modules/varnish/manifests/instance.pp-                description  => "Check Varnish UDS ${uds_path}",
modules/varnish/manifests/instance.pp-                nrpe_command => "/usr/local/lib/nagios/plugins/check_varnish_uds --socket ${uds_path}",
modules/varnish/manifests/instance.pp-                sudo_user    => 'root',
modules/varnish/manifests/instance.pp-                notes_url    => 'https://wikitech.wikimedia.org/wiki/Varnish',
modules/varnish/manifests/instance.pp-                require      => File['/usr/local/lib/nagios/plugins/check_varnish_uds'],
modules/varnish/manifests/instance.pp-            }

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Jan 22 2025, 2:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 22 2025, 2:15 PM
fgiunchedi mentioned this in T350360: Evaluate "drop in" replacement for nrpe scripts.Jan 22 2025, 2:15 PM
jcrespo subscribed.Jan 22 2025, 2:20 PM

Note check_bacula.py doesn't require porting, when creating it I implemented both prometheus scrapping and icinga functionalities for backwards compatibility (alertmanager wasn't available back then).

fgiunchedi added a comment.Jan 23 2025, 8:57 AM
In T384472#10484112, @jcrespo wrote:

Note check_bacula.py doesn't require porting, when creating it I implemented both prometheus scrapping and icinga functionalities for backwards compatibility (alertmanager wasn't available back then).

Fantastic, thank you that's excellent news in knowing we have one less thing to port!

fgiunchedi updated the task description. (Show Details)Jan 23 2025, 9:54 AM
tappof mentioned this in T395446: Evaluate which solution we could adopt as a drop-in replacement for NRPE (and start prototyping).May 28 2025, 1:07 PM
tappof triaged this task as Medium priority.Jun 24 2025, 1:08 PM
jcrespo added a comment.Jul 10 2025, 10:15 AM

Re: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1167691

So my next steps would be to implement a "nrpe::monitor_service"-compatible puppet definition (for easier migration) that 1) gets collected on the local host and sets up the config and run of the systemd service/systemd timer (similar to modules/nrpe/manifests/check.pp) and 2) setups up a scrapping job on the right prometheus hosts (similar to modules/icinga/manifests/naggen.pp) and 3) implements an alert on alertmanager.

Again, the idea is that this would be still a prototype to validate the architecture and needs, and later we can decide if this is how we want to take on these lose checks; and refine all the hard requirments before full production.

tappof updated the task description. (Show Details)Sep 3 2025, 7:27 AM
tappof claimed this task.Sep 3 2025, 7:29 AM
gerritbot added a comment.Sep 3 2025, 7:36 AM

Change #1184469 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] check_gdnsd_checkconf: enable nrpe wrapper

https://gerrit.wikimedia.org/r/1184469

gerritbot added a project: Patch-For-Review.Sep 3 2025, 7:36 AM
tappof updated the task description. (Show Details)Sep 3 2025, 7:46 AM
gerritbot added a comment.Oct 9 2025, 9:40 AM

Change #1184469 merged by Tiziano Fogli:

[operations/puppet@production] check_gdnsd_checkconf: enable nrpe wrapper

https://gerrit.wikimedia.org/r/1184469

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2025, 10:32 AM
gerritbot added a comment.Oct 17 2025, 1:57 PM

Change #1196918 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] haproxy_alive: enable nrpe wrapper

https://gerrit.wikimedia.org/r/1196918

gerritbot added a project: Patch-For-Review.Oct 17 2025, 1:57 PM
gerritbot added a comment.Oct 17 2025, 2:33 PM

Change #1196925 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] haproxy_failover: enable nrpe wrapper

https://gerrit.wikimedia.org/r/1196925

tappof updated the task description. (Show Details)Oct 17 2025, 3:50 PM
tappof updated the task description. (Show Details)Oct 17 2025, 3:54 PM
gerritbot added a comment.Oct 17 2025, 4:08 PM

Change #1196939 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] dbbackups: enable nrpe2nodexp wrapper on mariadb_${type}_... checks

https://gerrit.wikimedia.org/r/1196939

tappof updated the task description. (Show Details)Oct 17 2025, 4:13 PM
gerritbot added a comment.Oct 17 2025, 4:25 PM

Change #1196943 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] monitoring: enable nrpe2nodexp wrapper on <dir>_owned

https://gerrit.wikimedia.org/r/1196943

tappof updated the task description. (Show Details)Oct 17 2025, 4:26 PM
gerritbot added a comment.Oct 20 2025, 4:32 PM

Change #1196918 merged by Tiziano Fogli:

[operations/puppet@production] haproxy: enable nrpe2nodexp wrapper on haproxy_alive check

https://gerrit.wikimedia.org/r/1196918

gerritbot added a comment.Oct 20 2025, 4:33 PM

Change #1196925 merged by Tiziano Fogli:

[operations/puppet@production] mariadb::proxy::master: enable nrpe2ndoexp wrapper on haproxy_failover

https://gerrit.wikimedia.org/r/1196925

gerritbot added a comment.Oct 20 2025, 4:40 PM

Change #1196943 merged by Tiziano Fogli:

[operations/puppet@production] monitoring: enable nrpe2nodexp wrapper on <dir>_owned

https://gerrit.wikimedia.org/r/1196943

gerritbot added a comment.Oct 21 2025, 12:34 PM

Change #1196939 merged by Tiziano Fogli:

[operations/puppet@production] dbbackups: enable nrpe2nodexp wrapper on mariadb_${type}_... checks

https://gerrit.wikimedia.org/r/1196939

Maintenance_bot removed a project: Patch-For-Review.Oct 21 2025, 1:31 PM
tappof changed the subtype of this task from "Task" to "Goal".Oct 28 2025, 11:16 AM
tappof updated the task description. (Show Details)Oct 28 2025, 11:25 AM
tappof updated the task description. (Show Details)Oct 28 2025, 11:28 AM
gerritbot added a comment.Oct 30 2025, 9:48 AM

Change #1200012 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] haproxy: enable nrpe2nodexp wrapper on check-cinder-snapshot-leaks

https://gerrit.wikimedia.org/r/1200012

gerritbot added a project: Patch-For-Review.Oct 30 2025, 9:48 AM
tappof updated the task description. (Show Details)Oct 30 2025, 10:16 AM
gerritbot added a comment.Oct 30 2025, 10:19 AM

Change #1200016 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] neutron: enable nrpe2nodexp wrapper on check-neutron-conntrack

https://gerrit.wikimedia.org/r/1200016

tappof updated the task description. (Show Details)Oct 30 2025, 10:20 AM
gerritbot added a comment.Oct 30 2025, 10:23 AM

Change #1200018 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] nova: enable nrpe2nodexp wrapper on check-flavor_aggregates

https://gerrit.wikimedia.org/r/1200018

tappof updated the task description. (Show Details)Oct 30 2025, 10:23 AM
tappof updated the task description. (Show Details)Oct 30 2025, 10:39 AM
gerritbot added a comment.Oct 30 2025, 2:02 PM

Change #1200012 merged by Tiziano Fogli:

[operations/puppet@production] cinder: enable nrpe2nodexp wrapper on check-cinder-snapshot-leaks

https://gerrit.wikimedia.org/r/1200012

gerritbot added a comment.Oct 30 2025, 2:13 PM

Change #1200074 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] dbctl: enable nrpe2nodexp wrapper on dbctl_uncommitted_diffs

https://gerrit.wikimedia.org/r/1200074

tappof updated the task description. (Show Details)Oct 30 2025, 2:35 PM
gerritbot added a comment.Oct 30 2025, 3:01 PM

Change #1200088 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] dotls: enable nrpe2nodexp wrapper on check_dotls

https://gerrit.wikimedia.org/r/1200088

gerritbot added a comment.Oct 30 2025, 3:06 PM

Change #1200074 merged by Tiziano Fogli:

[operations/puppet@production] dbctl: enable nrpe2nodexp wrapper on dbctl_uncommitted_diffs

https://gerrit.wikimedia.org/r/1200074

tappof updated the task description. (Show Details)Oct 30 2025, 3:07 PM
gerritbot added a comment.Oct 31 2025, 2:19 PM

Change #1200359 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] dns: enable nrpe2nodexp wrapper on authdns_update_run check

https://gerrit.wikimedia.org/r/1200359

tappof updated the task description. (Show Details)Oct 31 2025, 2:21 PM
tappof updated the task description. (Show Details)
tappof updated the task description. (Show Details)Oct 31 2025, 2:24 PM
gerritbot added a comment.Oct 31 2025, 2:34 PM

Change #1200362 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] lvs: enable nrpe2nodexp wrapper on check_rp_filter_disabled check

https://gerrit.wikimedia.org/r/1200362

tappof updated the task description. (Show Details)Oct 31 2025, 2:41 PM
tappof updated the task description. (Show Details)
gerritbot added a comment.Oct 31 2025, 2:48 PM

Change #1200365 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] netbox: enable nrpe2nodexp wrapper on check_uncommitted_dns_changes check

https://gerrit.wikimedia.org/r/1200365

tappof updated the task description. (Show Details)Oct 31 2025, 2:56 PM
gerritbot added a comment.Nov 12 2025, 1:28 PM

Change #1200365 merged by Tiziano Fogli:

[operations/puppet@production] netbox: enable nrpe2nodexp wrapper on check_uncommitted_dns_changes check

https://gerrit.wikimedia.org/r/1200365

gerritbot added a comment.Nov 12 2025, 1:28 PM

Change #1200016 merged by Tiziano Fogli:

[operations/puppet@production] neutron: enable nrpe2nodexp wrapper on check-neutron-conntrack

https://gerrit.wikimedia.org/r/1200016

gerritbot added a comment.Nov 12 2025, 1:29 PM

Change #1200018 merged by Tiziano Fogli:

[operations/puppet@production] nova: enable nrpe2nodexp wrapper on check-flavor_aggregates

https://gerrit.wikimedia.org/r/1200018

gerritbot added a comment.Nov 12 2025, 1:29 PM

Change #1200088 merged by Tiziano Fogli:

[operations/puppet@production] dotls: enable nrpe2nodexp wrapper on check_dotls

https://gerrit.wikimedia.org/r/1200088

gerritbot added a comment.Nov 12 2025, 1:29 PM

Change #1200359 merged by Tiziano Fogli:

[operations/puppet@production] dns: enable nrpe2nodexp wrapper on authdns_update_run check

https://gerrit.wikimedia.org/r/1200359

tappof added a subtask: T413743: Alert in need of triage: nrpe_Check_whether_ferm_is_active_by_checking_the_default_input_chain (instance cumin2002:9100).Jan 14 2026, 1:40 PM
tappof added a subtask: T413744: Alert in need of triage: nrpe_Check_whether_ferm_is_active_by_checking_the_default_input_chain (instance cloudidp2001-dev:9100).
tappof added a subtask: T413745: Alert in need of triage: nrpe_Check_whether_ferm_is_active_by_checking_the_default_input_chain (instance cloudcumin2001:9100).
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits