Page MenuHomePhabricator

Port or deprecate pybal Icinga checks
Open, Needs TriagePublicGoal

Description

We have the following icinga checks related to pybal:

class profile::pybal(
    nrpe::monitor_service { 'check_service_restart_pybal':
        description    => 'Check if Pybal has been restarted after pybal.conf was changed',
        nrpe_command   => '/usr/local/lib/nagios/plugins/check_pybal_restart --service pybal.service --file /etc/pybal/pybal.conf',
        check_interval => 120, # 120mins
        retry_interval => 60,  # 60mins
        notes_url      => 'https://wikitech.wikimedia.org/wiki/PyBal#Pybal_service_has_not_been_restarted',
    }
class pybal::monitoring(
    nrpe::plugin { 'check_pybal':
        source => 'puppet:///modules/pybal/check_pybal',
    } 
      
    nrpe::monitor_service { 'pybal_backends':
        description  => 'PyBal backends health check',
        nrpe_command => '/usr/local/lib/nagios/plugins/check_pybal --url http://localhost:9090/alerts',
        notes_url    => 'https://wikitech.wikimedia.org/wiki/PyBal',
    } 
      
    nrpe::plugin { 'check_pybal_ipvs_diff':
        source => 'puppet:///modules/pybal/check_pybal_ipvs_diff.py',
    } 

    nrpe::monitor_service { 'pybal_ipvs_diff':
        description    => 'PyBal IPVS diff check',
        nrpe_command   => "/usr/local/lib/nagios/plugins/check_pybal_ipvs_diff --req-timeout=10.0 --prometheus-url http://${::ipaddress}:9100/metrics",
        check_interval => 5,
        timeout        => 60,
        notes_url      => 'https://wikitech.wikimedia.org/wiki/PyBal',
    } 
    if $config_source == 'etcd' {
        # Get the configuration of all services for this LVS host
        # then sum all values.
        $n_etcd_connections = map($services) |$name,$service| {
            size($service['ip'][$::site])
        }.reduce() |$memo,$value| { $memo + $value }
      
        nrpe::monitor_service { 'pybal_etcd_connections':
            description    => 'PyBal connections to etcd',
            nrpe_command   => "/usr/local/lib/nagios/plugins/check_established_connections ${config_host} ${etcd_port} ${n_etcd_connections}",
            check_interval => 5,
            timeout        => 60,
            notes_url      => 'https://wikitech.wikimedia.org/wiki/PyBal',
        }
    }

The purpose of this task is to assess whether we have enough information in Prometheus to move alerts there, the alerts are still relevant, or whether we're keeping the checks as is (with compat layer described in T350360)

! MIGRATION TABLE !

Migrated? (Y/N)TitleResource TypeCommandFileProfiles
Npybal_ipvs_diffNrpe::Monitor_service/usr/local/lib/nagios/plugins/check_pybal_ipvs_diffmodules/pybal/manifests/monitoring.pp:36profile::pybal
Npybal_backendsNrpe::Monitor_service/usr/local/lib/nagios/plugins/check_pybalmodules/pybal/manifests/monitoring.pp:25profile::pybal
Ncheck_service_restart_pybalNrpe::Monitor_service/usr/local/lib/nagios/plugins/check_pybal_restartmodules/profile/manifests/pybal.pp:89profile::pybal
Npybal_etcd_connectionsNrpe::Monitor_service/usr/local/lib/nagios/plugins/check_established_connectionsmodules/pybal/manifests/monitoring.pp:52profile::pybal

Event Timeline

Change #1155251 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] monitoring services: add migration task T384938 to instances

https://gerrit.wikimedia.org/r/1155251

Change #1155251 merged by Tiziano Fogli:

[operations/puppet@production] monitoring services: add migration task T384938 to instances

https://gerrit.wikimedia.org/r/1155251

tappof changed the subtype of this task from "Task" to "Goal".Sep 2 2025, 1:34 PM