In the process of deprecating/removing icinga we have a few scripts invoked via nrpe (i.e. locally on the host, as opposed on the icinga host). It would be convenient to at least evaluate if we can ease the icinga nrpe migration story by being able to use the nrpe scripts more or less unchanged.
## Problem statement
Icinga checks can contain arbitrary logic since the execution model is "exec the program and alert based on its exit status". Over time we have written, deployed and tested checks with complex logic to ensure services work as expected.
As we migrate away from Icinga and onto Prometheus/Alertmanager some of these checks can be rewritten to be based on service metrics instead. However for some we can't do it now (e.g. no service metrics yet, no engineering resources, etc) and thus we have to support an "icinga compatibility layer".
In practical terms this means for puppet users to be able to run an icinga check on their hosts and get an alertmanager alert back if something goes wrong (i.e. exit status != 0). To support the current use cases the alert should also include the first line of stdout/stderr.
## Scope
Checks with significant and complex logic are in scope to be used by the compatibility layer. Simple checks or checks with modern replacements are not in scope. (e.g. `check_procs` nowadays is covered by systemd metrics and SystemdUnitFailed alert or `check_http` is covered by blackbox checks and their alerts)
## Implementation options
###1. Dedicated exporter
With this option we have the following:
1. https://github.com/canonical/nrpe_exporter deployed to the host, which talks to the local nrpe server
1. puppet builds the exporter configuration, configuring how to run each script
1. puppet configures prometheus to scrape said script
1. puppet deploys a corresponding prometheus alert for metrics
######Pros
1. nrpe exporter is already written and tested
1. the `nrpe::monitor_service` puppet code can stay in place as-is, since we would be essentially talking to nrpe-server instead of icinga
######Cons
1. nrpe-server keeps running on the hosts, which is literally remote command execution
1. command output is not reported in metrics and thus in the alert
1. we need to scrape (i.e. run the nrpe commands) at most every five minutes to have continuous metrics on prometheus
#####2. Using node-exporter textfile wrapper
With this solution we develop a wrapper to execute the given script and capture its exit status, and stdout/stderr. The metrics are written in Prometheus format on the filesystem for node-exporter to pick up. A systemd timer periodically executes the wrapper + script.
######Pros
1. nrpe is out of the picture
1. scripts can be executed with any cadence (i.e. not max every 5 minutes)
1. stdout/stderr can be included in alert text
######Cons
1. some (not a lot) code to be written and tested for said wrapper
1. more changes at puppet level
1. potentially high cardinality metrics if output changes often
## Candidates
The puppet snippets below are an audit (as of commit 36a2a9ea99e0ff) of nrpe checks using scripts without an immediate prometheus-ready replacement. Note that the usual disclaimer "some of these might be not relevant anymore" applies.
#### check_microcode
```
modules/base/manifests/kernel.pp: nrpe::monitor_service { 'cpu_microcode_status':
modules/base/manifests/kernel.pp- ensure => 'present',
modules/base/manifests/kernel.pp- description => 'Check whether microcode mitigations for CPU vulnerabilities are applied',
modules/base/manifests/kernel.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_microcode',
modules/base/manifests/kernel.pp- contact_group => 'admins',
modules/base/manifests/kernel.pp- check_interval => 1440,
modules/base/manifests/kernel.pp- retry_interval => 5,
modules/base/manifests/kernel.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Microcode',
modules/base/manifests/kernel.pp- }
modules/base/manifests/kernel.pp-
```
#### check_gdnsd_checkconf
```
modules/gdnsd/manifests/monitor_conf.pp: nrpe::monitor_service { 'gdnsd_checkconf':
modules/gdnsd/manifests/monitor_conf.pp- description => 'gdnsd checkconf',
modules/gdnsd/manifests/monitor_conf.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_gdnsd_checkconf',
modules/gdnsd/manifests/monitor_conf.pp- critical => true,
modules/gdnsd/manifests/monitor_conf.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/DNS#gdnsd_checkconf',
modules/gdnsd/manifests/monitor_conf.pp- }
```
#### check_haproxy
```
modules/haproxy/manifests/init.pp: nrpe::monitor_service { 'haproxy_alive':
modules/haproxy/manifests/init.pp- ensure => bool2str($monitor_check_haproxy, 'present', 'absent'),
modules/haproxy/manifests/init.pp- description => 'haproxy alive',
modules/haproxy/manifests/init.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_haproxy --check=alive',
modules/haproxy/manifests/init.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/HAProxy',
modules/haproxy/manifests/init.pp- }
```
```
modules/profile/manifests/mariadb/proxy/master.pp: nrpe::monitor_service { 'haproxy_failover':
modules/profile/manifests/mariadb/proxy/master.pp- description => 'haproxy failover',
modules/profile/manifests/mariadb/proxy/master.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_haproxy --check=failover',
modules/profile/manifests/mariadb/proxy/master.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/HAProxy',
modules/profile/manifests/mariadb/proxy/master.pp- }
```
#### check_cirrus_settings.py
```
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp: nrpe::monitor_service { "elasticsearch_setting_check_${port}":
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp- critical => false,
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp- contact_group => 'admins,team-discovery',
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Search#Administration',
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp- nrpe_command => "/usr/lib/nagios/plugins/check_cirrus_settings.py --url http://localhost:${port} --settings-file /etc/elasticsearch/${title}/cirrus_check_settings.yaml",
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp- description => "ElasticSearch setting check - ${port}",
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp- check_interval => 720, # 12h
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp- retry_interval => 120, # 2h
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp- retries => 1,
modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp- }
```
#### check-mariadb-backups
```
modules/dbbackups/manifests/check.pp: nrpe::monitor_service { "mariadb_${type}_${section}_${datacenter}":
modules/dbbackups/manifests/check.pp- description => "${type} of ${section} in ${datacenter}",
modules/dbbackups/manifests/check.pp- nrpe_command => $check_command,
modules/dbbackups/manifests/check.pp- critical => false,
modules/dbbackups/manifests/check.pp- contact_group => 'admins',
modules/dbbackups/manifests/check.pp- sudo_user => 'backupcheck',
modules/dbbackups/manifests/check.pp- check_interval => 30, # Don't check too often
modules/dbbackups/manifests/check.pp- require => [
modules/dbbackups/manifests/check.pp- Package['wmfbackups-check'],
modules/dbbackups/manifests/check.pp- File['/etc/wmfbackups/valid_sections.txt'],
modules/dbbackups/manifests/check.pp- User['backupcheck'],
modules/dbbackups/manifests/check.pp- ],
modules/dbbackups/manifests/check.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Rerun_a_failed_backup',
modules/dbbackups/manifests/check.pp- }
modules/dbbackups/manifests/check.pp-}
```
#### db-check-health
```
modules/mariadb/manifests/monitor_eventscheduler.pp: nrpe::monitor_service { "mariadb_event_scheduler_${name}":
modules/mariadb/manifests/monitor_eventscheduler.pp- description => "MariaDB Event Scheduler ${name}",
modules/mariadb/manifests/monitor_eventscheduler.pp- nrpe_command => $check_command,
modules/mariadb/manifests/monitor_eventscheduler.pp- critical => $is_critical,
modules/mariadb/manifests/monitor_eventscheduler.pp- contact_group => $contact_group,
modules/mariadb/manifests/monitor_eventscheduler.pp- require => Package['wmfmariadbpy-common'],
modules/mariadb/manifests/monitor_eventscheduler.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Event_Scheduler',
modules/mariadb/manifests/monitor_eventscheduler.pp- }
modules/mariadb/manifests/monitor_eventscheduler.pp-}
```
```
modules/mariadb/manifests/monitor_readonly.pp: nrpe::monitor_service { "mariadb_read_only_${name}":
modules/mariadb/manifests/monitor_readonly.pp- description => "MariaDB read only ${name}",
modules/mariadb/manifests/monitor_readonly.pp- nrpe_command => $check_command,
modules/mariadb/manifests/monitor_readonly.pp- critical => $is_critical,
modules/mariadb/manifests/monitor_readonly.pp- contact_group => $contact_group,
modules/mariadb/manifests/monitor_readonly.pp- require => Package['wmfmariadbpy-common'],
modules/mariadb/manifests/monitor_readonly.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Master_comes_back_in_read_only',
modules/mariadb/manifests/monitor_readonly.pp- }
modules/mariadb/manifests/monitor_readonly.pp-}
```
#### pmp-check-unix-memory
```
modules/mariadb/manifests/monitor_memory.pp: nrpe::monitor_service { 'mariadb_memory':
modules/mariadb/manifests/monitor_memory.pp- description => 'MariaDB memory',
modules/mariadb/manifests/monitor_memory.pp- nrpe_command => $check_command,
modules/mariadb/manifests/monitor_memory.pp- critical => $is_critical,
modules/mariadb/manifests/monitor_memory.pp- contact_group => $contact_group,
modules/mariadb/manifests/monitor_memory.pp- require => Package['monitoring-plugins-contrib'],
modules/mariadb/manifests/monitor_memory.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting',
modules/mariadb/manifests/monitor_memory.pp- }
modules/mariadb/manifests/monitor_memory.pp-}
```
#### check_mariadb
```
modules/mariadb/manifests/monitor_replication.pp: nrpe::monitor_service { "mariadb_replica_io_state_${name}":
modules/mariadb/manifests/monitor_replication.pp- description => "MariaDB Replica IO: ${name}",
modules/mariadb/manifests/monitor_replication.pp- nrpe_command => "${check_mariadb} --check=slave_io_state",
modules/mariadb/manifests/monitor_replication.pp- critical => $is_critical,
modules/mariadb/manifests/monitor_replication.pp- contact_group => $contact_group,
modules/mariadb/manifests/monitor_replication.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica',
modules/mariadb/manifests/monitor_replication.pp- }
modules/mariadb/manifests/monitor_replication.pp-
modules/mariadb/manifests/monitor_replication.pp: nrpe::monitor_service { "mariadb_replica_sql_state_${name}":
modules/mariadb/manifests/monitor_replication.pp- description => "MariaDB Replica SQL: ${name}",
modules/mariadb/manifests/monitor_replication.pp- nrpe_command => "${check_mariadb} --check=slave_sql_state",
modules/mariadb/manifests/monitor_replication.pp- critical => $is_critical,
modules/mariadb/manifests/monitor_replication.pp- contact_group => $contact_group,
modules/mariadb/manifests/monitor_replication.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica',
modules/mariadb/manifests/monitor_replication.pp- }
modules/mariadb/manifests/monitor_replication.pp-
modules/mariadb/manifests/monitor_replication.pp- # check the lag towards the $source_dc's master
modules/mariadb/manifests/monitor_replication.pp: nrpe::monitor_service { "mariadb_replica_sql_lag_${name}":
modules/mariadb/manifests/monitor_replication.pp- description => "MariaDB Replica Lag: ${name}",
modules/mariadb/manifests/monitor_replication.pp- nrpe_command => "${check_mariadb} --check=slave_sql_lag \
modules/mariadb/manifests/monitor_replication.pp- --shard=${name} --datacenter=${source_dc} \
modules/mariadb/manifests/monitor_replication.pp- --sql-lag-warn=${lag_warn} \
modules/mariadb/manifests/monitor_replication.pp- --sql-lag-crit=${lag_crit}",
modules/mariadb/manifests/monitor_replication.pp- retries => 10,
modules/mariadb/manifests/monitor_replication.pp- critical => $is_critical,
modules/mariadb/manifests/monitor_replication.pp- contact_group => $contact_group,
modules/mariadb/manifests/monitor_replication.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica',
modules/mariadb/manifests/monitor_replication.pp- }
modules/mariadb/manifests/monitor_replication.pp-}
```
#### check_dir-not-bad-owner
```
modules/monitoring/manifests/icinga/bad_directory_owner.pp: nrpe::monitor_service { "${safe_title}_owned":
modules/monitoring/manifests/icinga/bad_directory_owner.pp- description => "Improperly owned (${uid}:${gid}) files in ${title}",
modules/monitoring/manifests/icinga/bad_directory_owner.pp- nrpe_command => $filename,
modules/monitoring/manifests/icinga/bad_directory_owner.pp- check_interval => $interval,
modules/monitoring/manifests/icinga/bad_directory_owner.pp- timeout => $timeout,
modules/monitoring/manifests/icinga/bad_directory_owner.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Monitoring/bad_directory_owner',
modules/monitoring/manifests/icinga/bad_directory_owner.pp- }
modules/monitoring/manifests/icinga/bad_directory_owner.pp-}
```
#### check_cinder_snapshot_leaks.py
```
modules/openstack/manifests/cinder/monitor.pp: nrpe::monitor_service { 'check-cinder-snapshot-leaks':
modules/openstack/manifests/cinder/monitor.pp- ensure => 'present',
modules/openstack/manifests/cinder/monitor.pp- nrpe_command => '/usr/local/bin/check_cinder_snapshot_leaks.py',
modules/openstack/manifests/cinder/monitor.pp- description => 'Check for snapshots leaked by cinder backup agent',
modules/openstack/manifests/cinder/monitor.pp- require => File['/usr/local/bin/check_cinder_snapshot_leaks.py'],
modules/openstack/manifests/cinder/monitor.pp- contact_group => 'wmcs-team-email,admins',
modules/openstack/manifests/cinder/monitor.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_for_snapshots_leaked_by_cinder_backup_agent',
modules/openstack/manifests/cinder/monitor.pp- }
```
#### check_neutron_conntrack
```
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp: nrpe::monitor_service { 'check-neutron-conntrack':
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp- ensure => 'present',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_neutron_conntrack',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp- sudo_user => 'root',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp- description => 'Check nf_conntrack usage in neutron netns',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp- contact_group => 'wmcs-team-email,admins',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting',
modules/openstack/manifests/monitor/neutron/l3_agent_conntrack.pp- }
```
#### check_flavor_properties
```
modules/openstack/manifests/nova/fullstack/monitor.pp: nrpe::monitor_service { 'check-flavor_aggregates':
modules/openstack/manifests/nova/fullstack/monitor.pp- ensure => 'present',
modules/openstack/manifests/nova/fullstack/monitor.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_flavor_properties',
modules/openstack/manifests/nova/fullstack/monitor.pp- sudo_user => 'root',
modules/openstack/manifests/nova/fullstack/monitor.pp- description => 'all nova flavors are assigned necessary properties',
modules/openstack/manifests/nova/fullstack/monitor.pp- timeout => 30,
modules/openstack/manifests/nova/fullstack/monitor.pp- check_interval => 15,
modules/openstack/manifests/nova/fullstack/monitor.pp- contact_group => 'wmcs-team-email,wmcs-bots',
modules/openstack/manifests/nova/fullstack/monitor.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Host_aggregates';
modules/openstack/manifests/nova/fullstack/monitor.pp- }
```
#### check_bacula.py
```
modules/profile/manifests/backup/director.pp: nrpe::monitor_service { 'backup_freshness':
modules/profile/manifests/backup/director.pp- description => 'Backup freshness',
modules/profile/manifests/backup/director.pp- nrpe_command => '/usr/bin/check_bacula.py --icinga',
modules/profile/manifests/backup/director.pp- sudo_user => 'bacula',
modules/profile/manifests/backup/director.pp- critical => false,
modules/profile/manifests/backup/director.pp- contact_group => 'admins',
modules/profile/manifests/backup/director.pp- check_interval => 60, # check every hour
modules/profile/manifests/backup/director.pp- timeout => 60, # 1 minute of timeout, the check is not fast
modules/profile/manifests/backup/director.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Bacula#Monitoring',
modules/profile/manifests/backup/director.pp- }
```
#### check_dbctl_uncommitted_diffs
```
modules/profile/manifests/conftool/dbctl_client.pp: nrpe::monitor_service { 'dbctl_uncommitted_diffs':
modules/profile/manifests/conftool/dbctl_client.pp- ensure => present,
modules/profile/manifests/conftool/dbctl_client.pp- description => 'Uncommitted dbctl configuration changes, check dbctl config diff',
modules/profile/manifests/conftool/dbctl_client.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_dbctl_uncommitted_diffs',
modules/profile/manifests/conftool/dbctl_client.pp- critical => false,
modules/profile/manifests/conftool/dbctl_client.pp- check_interval => 5,
modules/profile/manifests/conftool/dbctl_client.pp- retry_interval => 5,
modules/profile/manifests/conftool/dbctl_client.pp- retries => 3,
modules/profile/manifests/conftool/dbctl_client.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Dbctl#Uncommitted_dbctl_diffs',
modules/profile/manifests/conftool/dbctl_client.pp- timeout => 20,
modules/profile/manifests/conftool/dbctl_client.pp- }
```
#### check_dotls
```
modules/profile/manifests/dns/auth/dotls.pp: nrpe::monitor_service { 'check_dotls':
modules/profile/manifests/dns/auth/dotls.pp- description => 'AuthDNS-over-TLS Works',
modules/profile/manifests/dns/auth/dotls.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_dotls',
modules/profile/manifests/dns/auth/dotls.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/DNS',
modules/profile/manifests/dns/auth/dotls.pp- }
```
#### check_authdns_update_run
```
modules/profile/manifests/dns/auth/update.pp: nrpe::monitor_service { 'authdns_update_run':
modules/profile/manifests/dns/auth/update.pp- description => 'check if authdns-update was run after a change was merged to operations/dns.git',
modules/profile/manifests/dns/auth/update.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_authdns_update_run',
modules/profile/manifests/dns/auth/update.pp- check_interval => 5, # min
modules/profile/manifests/dns/auth/update.pp- retry_interval => 5, # min
modules/profile/manifests/dns/auth/update.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/DNS#authdns_update_run',
modules/profile/manifests/dns/auth/update.pp- }
```
#### check_ferm
```
modules/profile/manifests/firewall.pp: nrpe::monitor_service { 'ferm_active':
modules/profile/manifests/firewall.pp- description => 'Check whether ferm is active by checking the default input chain',
modules/profile/manifests/firewall.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_ferm',
modules/profile/manifests/firewall.pp- sudo_user => 'root',
modules/profile/manifests/firewall.pp- contact_group => 'admins',
modules/profile/manifests/firewall.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm',
modules/profile/manifests/firewall.pp- check_interval => 30,
modules/profile/manifests/firewall.pp- retry_interval => $ferm_icinga_retry_interval,
modules/profile/manifests/firewall.pp- }
```
#### check_sysctl
```
modules/profile/manifests/lvs.pp: nrpe::monitor_service { 'check_rp_filter_disabled':
modules/profile/manifests/lvs.pp- description => 'Check rp_filter disabled',
modules/profile/manifests/lvs.pp- nrpe_command => "/usr/local/lib/nagios/plugins/check_sysctl ${rp_args}",
modules/profile/manifests/lvs.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Monitoring/check_rp_filter_disabled',
modules/profile/manifests/lvs.pp- }
```
#### check_exim_queue
```
modules/profile/manifests/mail/smarthost.pp: nrpe::monitor_service { 'check_exim_queue':
modules/profile/manifests/mail/smarthost.pp- description => 'exim queue',
modules/profile/manifests/mail/smarthost.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_exim_queue -w 1000 -c 3000',
modules/profile/manifests/mail/smarthost.pp- check_interval => 30,
modules/profile/manifests/mail/smarthost.pp- retry_interval => 10,
modules/profile/manifests/mail/smarthost.pp- timeout => 20,
modules/profile/manifests/mail/smarthost.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Mail#Troubleshooting',
modules/profile/manifests/mail/smarthost.pp- }
```
#### check_json_file
```
modules/profile/manifests/netbox/automation.pp: nrpe::monitor_service { 'check_uncommitted_dns_changes':
modules/profile/manifests/netbox/automation.pp- ensure => $active_ensure,
modules/profile/manifests/netbox/automation.pp- check_interval => 5,
modules/profile/manifests/netbox/automation.pp- retry_interval => 2,
modules/profile/manifests/netbox/automation.pp- description => 'Uncommitted DNS changes in Netbox',
modules/profile/manifests/netbox/automation.pp- nrpe_command => "${check_command} ${icinga_state_file} ${max_age}",
modules/profile/manifests/netbox/automation.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Monitoring/Netbox_DNS_uncommitted_changes',
modules/profile/manifests/netbox/automation.pp- contact_group => 'team-dcops',
modules/profile/manifests/netbox/automation.pp- }
```
#### check_service_restart
```
modules/profile/manifests/ntp.pp: nrpe::monitor_service { "check_service_restart_${service}":
modules/profile/manifests/ntp.pp- description => "Check if ${service} has been restarted after ${conf_file} was changed",
modules/profile/manifests/ntp.pp- nrpe_command => "/usr/local/lib/nagios/plugins/check_ntp_service --service ${service} --file ${conf_file} --critical 2",
modules/profile/manifests/ntp.pp- sudo_user => 'root',
modules/profile/manifests/ntp.pp- check_interval => 60, # 60mins
modules/profile/manifests/ntp.pp- retry_interval => 30, # 30mins
modules/profile/manifests/ntp.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/NTP#Monitoring',
modules/profile/manifests/ntp.pp- }
```
```
modules/profile/manifests/pybal.pp: nrpe::monitor_service { 'check_service_restart_pybal':
modules/profile/manifests/pybal.pp- description => 'Check if Pybal has been restarted after pybal.conf was changed',
modules/profile/manifests/pybal.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_pybal_restart --service pybal.service --file /etc/pybal/pybal.conf',
modules/profile/manifests/pybal.pp- check_interval => 120, # 120mins
modules/profile/manifests/pybal.pp- retry_interval => 60, # 60mins
modules/profile/manifests/pybal.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/PyBal#Pybal_service_has_not_been_restarted',
modules/profile/manifests/pybal.pp- }
```
```
modules/profile/manifests/wikidough.pp: nrpe::monitor_service { "check_service_restart_${service}":
modules/profile/manifests/wikidough.pp- description => "Check if ${service} has been restarted after ${conf_file} was changed",
modules/profile/manifests/wikidough.pp- nrpe_command => "/usr/local/lib/nagios/plugins/check_wikidough_restart --service ${service} --file ${conf_file}",
modules/profile/manifests/wikidough.pp- sudo_user => 'root',
modules/profile/manifests/wikidough.pp- check_interval => 360, # 6h
modules/profile/manifests/wikidough.pp- retry_interval => 60, # 1h
modules/profile/manifests/wikidough.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring#Service_Restart_Check',
modules/profile/manifests/wikidough.pp- }
```
#### check_timedatectl
```
modules/profile/manifests/systemd/timesyncd.pp: nrpe::monitor_service { 'timesynd_ntp_status':
modules/profile/manifests/systemd/timesyncd.pp- ensure => $ensure,
modules/profile/manifests/systemd/timesyncd.pp- description => 'Check the NTP synchronisation status of timesyncd',
modules/profile/manifests/systemd/timesyncd.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_timedatectl',
modules/profile/manifests/systemd/timesyncd.pp- contact_group => 'admins',
modules/profile/manifests/systemd/timesyncd.pp- check_interval => 30,
modules/profile/manifests/systemd/timesyncd.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/NTP',
modules/profile/manifests/systemd/timesyncd.pp- }
```
#### check_default_ats_lua_conf
```
modules/profile/manifests/trafficserver/backend.pp: nrpe::monitor_service { 'default_ats_lua_conf':
modules/profile/manifests/trafficserver/backend.pp- description => 'Default ATS Lua configuration file',
modules/profile/manifests/trafficserver/backend.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_default_ats_lua_conf',
modules/profile/manifests/trafficserver/backend.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/ATS',
modules/profile/manifests/trafficserver/backend.pp- }
```
#### check_journal_pattern
```
modules/profile/manifests/trafficserver/logs.pp: nrpe::monitor_service { "${service_name}_skipped_logs":
modules/profile/manifests/trafficserver/logs.pp- description => "Logs skipped by ${service_name}",
modules/profile/manifests/trafficserver/logs.pp- nrpe_command => "/usr/local/lib/nagios/plugins/check_journal_pattern '1 hour ago' 'NOTE: Skipping the current log entry for ' ${service_name}",
modules/profile/manifests/trafficserver/logs.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/ATS',
modules/profile/manifests/trafficserver/logs.pp- }
```
#### check_pybal
```
modules/pybal/manifests/monitoring.pp: nrpe::monitor_service { 'pybal_backends':
modules/pybal/manifests/monitoring.pp- description => 'PyBal backends health check',
modules/pybal/manifests/monitoring.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_pybal --url http://localhost:9090/alerts',
modules/pybal/manifests/monitoring.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/PyBal',
modules/pybal/manifests/monitoring.pp- }
```
#### check_pybal_ipvs_diff
```
modules/pybal/manifests/monitoring.pp: nrpe::monitor_service { 'pybal_ipvs_diff':
modules/pybal/manifests/monitoring.pp- description => 'PyBal IPVS diff check',
modules/pybal/manifests/monitoring.pp- nrpe_command => "/usr/local/lib/nagios/plugins/check_pybal_ipvs_diff --req-timeout=10.0 --prometheus-url http://${::ipaddress}:9100/metrics",
modules/pybal/manifests/monitoring.pp- check_interval => 5,
modules/pybal/manifests/monitoring.pp- timeout => 60,
modules/pybal/manifests/monitoring.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/PyBal',
modules/pybal/manifests/monitoring.pp- }
```
#### check_established_connections
```
modules/pybal/manifests/monitoring.pp: nrpe::monitor_service { 'pybal_etcd_connections':
modules/pybal/manifests/monitoring.pp- description => 'PyBal connections to etcd',
modules/pybal/manifests/monitoring.pp- nrpe_command => "/usr/local/lib/nagios/plugins/check_established_connections ${config_host} ${etcd_port} ${n_etcd_connections}",
modules/pybal/manifests/monitoring.pp- check_interval => 5,
modules/pybal/manifests/monitoring.pp- timeout => 60,
modules/pybal/manifests/monitoring.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/PyBal',
modules/pybal/manifests/monitoring.pp- }
```
#### check_hpssacli
```
modules/raid/manifests/hpsa/hpssacli.pp: nrpe::monitor_service { 'raid_hpssacli':
modules/raid/manifests/hpsa/hpssacli.pp- description => 'HP RAID',
modules/raid/manifests/hpsa/hpssacli.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_hpssacli',
modules/raid/manifests/hpsa/hpssacli.pp- timeout => 90, # can take > 10s on servers with lots of disks
modules/raid/manifests/hpsa/hpssacli.pp- check_interval => $raid::check_interval,
modules/raid/manifests/hpsa/hpssacli.pp- retry_interval => $raid::retry_interval,
modules/raid/manifests/hpsa/hpssacli.pp- event_handler => "raid_handler!hpssacli!${::site}",
modules/raid/manifests/hpsa/hpssacli.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook#Hardware_Raid_Information_Gathering',
modules/raid/manifests/hpsa/hpssacli.pp- }
```
#### check_ssacli
```
modules/raid/manifests/hpsa/ssacli.pp: nrpe::monitor_service { 'raid_ssacli':
modules/raid/manifests/hpsa/ssacli.pp- description => 'HP RAID',
modules/raid/manifests/hpsa/ssacli.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_ssacli',
modules/raid/manifests/hpsa/ssacli.pp- timeout => 90, # can take > 10s on servers with lots of disks
modules/raid/manifests/hpsa/ssacli.pp- check_interval => $raid::check_interval,
modules/raid/manifests/hpsa/ssacli.pp- retry_interval => $raid::retry_interval,
modules/raid/manifests/hpsa/ssacli.pp- event_handler => "raid_handler!ssacli!${::site}",
modules/raid/manifests/hpsa/ssacli.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook#Hardware_Raid_Information_Gathering',
modules/raid/manifests/hpsa/ssacli.pp- }
```
#### check_megacli
```
modules/raid/manifests/megaraid.pp: nrpe::monitor_service { 'raid_megaraid':
modules/raid/manifests/megaraid.pp- description => 'MegaRAID',
modules/raid/manifests/megaraid.pp- nrpe_command => "${raid::check_raid} megacli",
modules/raid/manifests/megaraid.pp- sudo_user => 'root',
modules/raid/manifests/megaraid.pp- check_interval => $raid::check_interval,
modules/raid/manifests/megaraid.pp- retry_interval => $raid::retry_interval,
modules/raid/manifests/megaraid.pp- event_handler => "raid_handler!megacli!${::site}",
modules/raid/manifests/megaraid.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/MegaCli#Monitoring',
modules/raid/manifests/megaraid.pp- }
```
#### check_perccli
```
modules/raid/manifests/perccli.pp: nrpe::monitor_service { 'raid_perc_raid':
modules/raid/manifests/perccli.pp- description => 'Dell PowerEdge RAID Controller',
modules/raid/manifests/perccli.pp- nrpe_command => '/usr/local/lib/nagios/plugins/get-raid-status-perccli',
modules/raid/manifests/perccli.pp- sudo_user => 'root',
modules/raid/manifests/perccli.pp- check_interval => $raid::check_interval,
modules/raid/manifests/perccli.pp- retry_interval => $raid::retry_interval,
modules/raid/manifests/perccli.pp- event_handler => "raid_handler!perccli!${::site}",
modules/raid/manifests/perccli.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/PERCCli#Monitoring',
modules/raid/manifests/perccli.pp- }
```
#### check_udp2log_log_age
```
modules/udp2log/manifests/instance/monitoring.pp: nrpe::monitor_service { "udp2log_log_age-${name}":
modules/udp2log/manifests/instance/monitoring.pp- ensure => 'present',
modules/udp2log/manifests/instance/monitoring.pp- description => "udp2log log age for ${name}",
modules/udp2log/manifests/instance/monitoring.pp- nrpe_command => "/usr/lib/nagios/plugins/check_udp2log_log_age ${name}",
modules/udp2log/manifests/instance/monitoring.pp- contact_group => 'admins,analytics,team-data-platform',
modules/udp2log/manifests/instance/monitoring.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Udp2log',
modules/udp2log/manifests/instance/monitoring.pp- }
```
#### check_vcl_reload
```
modules/varnish/manifests/common/director_scripts.pp: nrpe::monitor_service { 'confd_vcl_reload':
modules/varnish/manifests/common/director_scripts.pp- description => 'Confd vcl based reload',
modules/varnish/manifests/common/director_scripts.pp- nrpe_command => '/usr/local/lib/nagios/plugins/check_vcl_reload',
modules/varnish/manifests/common/director_scripts.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Varnish',
modules/varnish/manifests/common/director_scripts.pp- }
```
#### check_varnish_uds
```
modules/varnish/manifests/instance.pp: nrpe::monitor_service { "check-varnish-uds${instancesuffix}-${uds_path}":
modules/varnish/manifests/instance.pp- ensure => present,
modules/varnish/manifests/instance.pp- description => "Check Varnish UDS ${uds_path}",
modules/varnish/manifests/instance.pp- nrpe_command => "/usr/local/lib/nagios/plugins/check_varnish_uds --socket ${uds_path}",
modules/varnish/manifests/instance.pp- sudo_user => 'root',
modules/varnish/manifests/instance.pp- notes_url => 'https://wikitech.wikimedia.org/wiki/Varnish',
modules/varnish/manifests/instance.pp- require => File['/usr/local/lib/nagios/plugins/check_varnish_uds'],
modules/varnish/manifests/instance.pp- }
```