Page MenuHomePhabricator

Port elasticsearch checks from Icinga to Prometheus/Alertmanager
Open, Needs TriagePublicGoal

Description

We have the following checks for elasticsearch in Icinga, which will need porting to Prometheus/Alertmanager. Note we do have opensearch alerts per T371083: Port profile::opensearch::monitoring::base_checks to Prometheus/Alertmanager which we can reuse/adapt for some of these too.

define icinga::monitor::elasticsearch::cirrus_checks(
    Enum['http', 'https'] $scheme = 'http',
    String $host = $::hostname,
    Array[Stdlib::Port] $ports = [9200],
    Integer $timeout = 4,
) {
    $ports.each |$port| {
        monitoring::service { "elasticsearch / cirrus frozen writes - ${host}:${port}":
            host          => $host,
            check_command => "check_cirrus_frozen_writes!${scheme}!${port}!${timeout}",
            description   => "ElasticSearch health check for frozen writes - ${port}",
            critical      => true,
            contact_group => 'admins,team-discovery',
            notes_url     => 'https://wikitech.wikimedia.org/wiki/Search#Pausing_Indexing',
        }

        monitoring::service { "elasticsearch / masters eligible - ${host}:${port}":
            host          => $host,
            check_command => "check_masters_eligible!${scheme}!${port}!${timeout}",
            description   => "ElasticSearch numbers of masters eligible - ${port}",
            critical      => false,
            contact_group => 'admins,team-discovery',
            notes_url     => 'https://wikitech.wikimedia.org/wiki/Search#Expected_eligible_masters_check_and_alert',
            retries       => 10,  # it is fine if we are missing a master for a short time (during reboots / restarts)
        }
    }
}
# = Define: icinga::monitor::elasticsearch::base_checks
define icinga::monitor::elasticsearch::base_checks(
    String $threshold = '>=0.2',
    Enum['http', 'https'] $scheme = 'http',
    String $host = $::hostname,
    Array[Stdlib::Port] $ports = [9200],
    Integer $shard_size_warning = 50,
    Integer $shard_size_critical = 60,
    Integer $timeout = 4,
    Boolean $use_nrpe = false,
) {
    $ports.each |$port| {
        # yes yes! a lot of duplication here which could be improved.
        # they will remain here until we find a better way
        # also always update both checks!
        if !$use_nrpe {
            monitoring::service {
                default:
                    host          => $host,
                    critical      => false,
                    contact_group => 'admins,team-discovery',
                    notes_url     => 'https://wikitech.wikimedia.org/wiki/Search#Administration',
                ;
                "elasticsearch_shards_${host}:${port}":
                    check_command => "check_elasticsearch_shards_threshold!${scheme}!${port}!${threshold}!${timeout}",
                    description   => "ElasticSearch health check for shards on ${port}",
                ;
                "elasticsearch_unassigned_shard_check_${host}:${port}":
                    check_command  => "check_elasticsearch_unassigned_shards!${scheme}!${port}!${timeout}",
                    description    => "ElasticSearch unassigned shard check - ${port}",
                    check_interval => 720, # 12h
                    retry_interval => 120, # 2h
                    retries        => 1,
                ;
                "elasticsearch_shard_size_check_${host}:${port}":
                    check_command  => "check_elasticsearch_shard_size!${scheme}!${port}!${shard_size_warning}!${shard_size_critical}!${timeout}",
                    description    => "ElasticSearch shard size check - ${port}",
                    check_interval => 1440, # 24h
                    retry_interval => 180, # 3h
                    notes_url      => 'https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed',
                ;
            }
        } else {
            require ::icinga::elasticsearch::base_plugin

            nrpe::monitor_service {
                default:
                    critical      => false,
                    contact_group => 'admins,team-discovery',
                    notes_url     => 'https://wikitech.wikimedia.org/wiki/Search#Administration',
                ;
                "elasticsearch_shards_${port}":
                    nrpe_command => "/usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:${port} --shards-inactive '${threshold}' --timeout ${timeout}",
                    description  => "ElasticSearch health check for shards on ${port}",
                ;
                "elasticsearch_unassigned_shard_check_${port}":
                    nrpe_command   => "/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:${port} --timeout ${timeout}",
                    description    => "ElasticSearch unassigned shard check - ${port}",
                    check_interval => 720, # 12h
                    retry_interval => 120, # 2h
                    retries        => 1,
                ;
                "elasticsearch_shard_size_check_${port}":
                    nrpe_command   => "/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:${port} --shard-size-warning ${shard_size_warning} --shard-size-critical ${shard_size_critical} --timeout ${timeout}",
                    description    => "ElasticSearch shard size check - ${port}",
                    check_interval => 1440, # 24h
                    retry_interval => 180, # 3h
                    notes_url      => 'https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed',
                ;
            }
        }
    }
}
define elasticsearch::tlsproxy (
    monitoring::service { "elasticsearch-https-${title}":
        ensure        => present,
        description   => "Elasticsearch HTTPS for ${title}",
        check_command => "${check_command}!${server_name}!${tls_port}",
        notes_url     => 'https://wikitech.wikimedia.org/wiki/Search',
    }

! MIGRATION TABLE !

Migrated? (Y/N)TitleResource TypeCommandFileProfiles
Nelasticsearch_unassigned_shard_check_search.svc.X.wmnet:XMonitoring::Servicecheck_elasticsearch_unassigned_shardsmodules/icinga/manifests/monitor/elasticsearch/base_checks.pp:17profile::icinga
Nelasticsearch_shards_search.svc.X.wmnet:XMonitoring::Servicecheck_elasticsearch_shards_thresholdmodules/icinga/manifests/monitor/elasticsearch/base_checks.pp:17profile::icinga
Nelasticsearch-https-XMonitoring::Servicecheck_ssl_on_portmodules/elasticsearch/manifests/tlsproxy.pp:45profile::opensearch::cirrus::server, profile::opensearch::cirrus::relforge
Nelasticsearch / masters eligible - search.svc.X.wmnet:XMonitoring::Servicecheck_masters_eligiblemodules/icinga/manifests/monitor/elasticsearch/cirrus_checks.pp:19profile::icinga
Nelasticsearch-https-cloudelastic-X-eqiad-XMonitoring::Servicecheck_ssl_on_port_letsencryptmodules/elasticsearch/manifests/tlsproxy.pp:45profile::opensearch::cirrus::server
Nelasticsearch_shard_size_check_search.svc.X.wmnet:XMonitoring::Servicecheck_elasticsearch_shard_sizemodules/icinga/manifests/monitor/elasticsearch/base_checks.pp:17profile::icinga

Event Timeline

Change #1155136 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] monitoring services: add migration task T384998 to instances

https://gerrit.wikimedia.org/r/1155136

Change #1155136 merged by Tiziano Fogli:

[operations/puppet@production] monitoring services: add migration task T384998 to instances

https://gerrit.wikimedia.org/r/1155136

tappof changed the subtype of this task from "Task" to "Goal".Sep 2 2025, 1:34 PM

Change #1318274 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrus: Remove Cirrus frozen write monitors

https://gerrit.wikimedia.org/r/1318274

Hey @tappof /0lly, just a heads-up that I removed the frozen writes check from your migration table, as it is checking a feature that was removed several years ago. Ref the attached CR if you would like more context or feel free to reach out on Slack/IRC.

Change #1318301 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: Replace icinga check with prometheus blackbox check

https://gerrit.wikimedia.org/r/1318301

Change #1318301 merged by Bking:

[operations/puppet@production] cirrussearch: Replace icinga check with prometheus blackbox check

https://gerrit.wikimedia.org/r/1318301

Change #1318274 merged by Bking:

[operations/puppet@production] cirrus: Remove Cirrus frozen write monitors

https://gerrit.wikimedia.org/r/1318274