We have the following icinga checks related to pybal:
class profile::pybal(
nrpe::monitor_service { 'check_service_restart_pybal':
description => 'Check if Pybal has been restarted after pybal.conf was changed',
nrpe_command => '/usr/local/lib/nagios/plugins/check_pybal_restart --service pybal.service --file /etc/pybal/pybal.conf',
check_interval => 120, # 120mins
retry_interval => 60, # 60mins
notes_url => 'https://wikitech.wikimedia.org/wiki/PyBal#Pybal_service_has_not_been_restarted',
}class pybal::monitoring(
nrpe::plugin { 'check_pybal':
source => 'puppet:///modules/pybal/check_pybal',
}
nrpe::monitor_service { 'pybal_backends':
description => 'PyBal backends health check',
nrpe_command => '/usr/local/lib/nagios/plugins/check_pybal --url http://localhost:9090/alerts',
notes_url => 'https://wikitech.wikimedia.org/wiki/PyBal',
}
nrpe::plugin { 'check_pybal_ipvs_diff':
source => 'puppet:///modules/pybal/check_pybal_ipvs_diff.py',
}
nrpe::monitor_service { 'pybal_ipvs_diff':
description => 'PyBal IPVS diff check',
nrpe_command => "/usr/local/lib/nagios/plugins/check_pybal_ipvs_diff --req-timeout=10.0 --prometheus-url http://${::ipaddress}:9100/metrics",
check_interval => 5,
timeout => 60,
notes_url => 'https://wikitech.wikimedia.org/wiki/PyBal',
}
if $config_source == 'etcd' {
# Get the configuration of all services for this LVS host
# then sum all values.
$n_etcd_connections = map($services) |$name,$service| {
size($service['ip'][$::site])
}.reduce() |$memo,$value| { $memo + $value }
nrpe::monitor_service { 'pybal_etcd_connections':
description => 'PyBal connections to etcd',
nrpe_command => "/usr/local/lib/nagios/plugins/check_established_connections ${config_host} ${etcd_port} ${n_etcd_connections}",
check_interval => 5,
timeout => 60,
notes_url => 'https://wikitech.wikimedia.org/wiki/PyBal',
}
}The purpose of this task is to assess whether we have enough information in Prometheus to move alerts there, the alerts are still relevant, or whether we're keeping the checks as is (with compat layer described in T350360)
! MIGRATION TABLE !
| Migrated? (Y/N) | Title | Resource Type | Command | File | Profiles |
|---|---|---|---|---|---|
| N | pybal_ipvs_diff | Nrpe::Monitor_service | /usr/local/lib/nagios/plugins/check_pybal_ipvs_diff | modules/pybal/manifests/monitoring.pp:36 | profile::pybal |
| N | pybal_backends | Nrpe::Monitor_service | /usr/local/lib/nagios/plugins/check_pybal | modules/pybal/manifests/monitoring.pp:25 | profile::pybal |
| N | check_service_restart_pybal | Nrpe::Monitor_service | /usr/local/lib/nagios/plugins/check_pybal_restart | modules/profile/manifests/pybal.pp:89 | profile::pybal |
| N | pybal_etcd_connections | Nrpe::Monitor_service | /usr/local/lib/nagios/plugins/check_established_connections | modules/pybal/manifests/monitoring.pp:52 | profile::pybal |