Page MenuHomePhabricator

Progress indicator for Icinga -> Alertmanager migration
Closed, ResolvedPublic

Description

To track the overall progress of Icinga -> AM migration I've come up with the following "progress indicator". Namely an aggregate list of all service checks we have in Icinga. Whenever we make progress (either removing checks altogether or moving checks to Alertmanager/Prometheus) the list will shrink.

In terms of targets, ideally the list would reach zero (i.e. no more checks in Icinga, we can shut it down) though that largely depends on a cost/benefit analysis since the tail of checks is quite long (subject for another task)

The following command will report on all puppet-generated checks in Icinga present at the moment of running, with some attempt at aggregating related checks. The total amount of services is also tracked in icinga_service_count metric.

grep _NAME /etc/icinga/objects/puppet_services.cfg | awk '{print $4}' | sort | sed -E -e 's@[a-z][-a-z0-9]+\.[a-z]+(\.[a-z]+)?@HOST@g' -e 's@(eqiad|codfw|esams|ulsfo|eqsin|drmrs)@SITE@g' -e 's@[0-9]{3,}@PORT@g' | sort | uniq -c | sort -nr

The results as of the writing of this task (20221017) can be found at P35498, listing 978 de-duplicated checks. For some of these work is already in progress for either decom'ing the check or porting to Prometheus (cfr T288622)

See also this spreadsheet: https://docs.google.com/spreadsheets/d/19nxCXldb804TJCXGy4Z2BHG_1wRksRnKcPC6sXfjQuM/edit#gid=1831147731
Icinga checks are tallied and deduplicated there, also annotated with their check "kind" (i.e. what we can do in a world without icinga (the engine/program)

Event Timeline

I've updated the previous command line which was misleading. The updated results are at P42248 and report 588 unique checks today

hi @fgiunchedi

Thanks for keeping the counts updated! how are we doing after sprint week?

thanks!

591 unique checks, though there's still a bunch to remove as sprint week followups!

580 unique checks as of today

🎉

I'm experimenting with a puppetdb-based audit/progress, so we can attach e.g. the profile to the check, for example:

# checks# profiles
486194
TitleProfileNotes
Auth DNS TCP: k8s.svc.tools.SITE1.wikimedia.cloud on server ns0.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS TCP: k8s.svc.tools.SITE1.wikimedia.cloud on server ns1.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS TCP: login.toolforge.org on server ns0.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS TCP: login.toolforge.org on server ns1.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS TCP: tools-sgegrid-master.tools.SITE1.wikimedia.cloud on server ns0.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS TCP: tools-sgegrid-master.tools.SITE1.wikimedia.cloud on server ns1.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS TCP: www.wmcloud.org on server ns0.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS TCP: www.wmcloud.org on server ns1.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS UDP: k8s.svc.tools.SITE1.wikimedia.cloud on server ns0.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS UDP: k8s.svc.tools.SITE1.wikimedia.cloud on server ns1.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS UDP: login.toolforge.org on server ns0.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS UDP: login.toolforge.org on server ns1.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS UDP: tools-sgegrid-master.tools.SITE1.wikimedia.cloud on server ns0.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS UDP: tools-sgegrid-master.tools.SITE1.wikimedia.cloud on server ns1.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS UDP: www.wmcloud.org on server ns0.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Auth DNS UDP: www.wmcloud.org on server ns1.openstack.SITE1.wikimediacloud.orgprofile::openstack::eqiad1::pdns::auth::service
Categories_Lagprofile::query_service::categories
Categories_Pingprofile::query_service::categories
DEVICE BFDprofile::icinga
DEVICE BGPprofile::icinga
DEVICE Juniperprofile::icinga
DEVICE OSPFprofile::icinga
DEVICE VCPprofile::icinga
DEVICE VRRPprofile::icinga
DEVICE interfacesprofile::icinga
DNS resolution k8s.svc.tools.SITE1.wikimedia.cloudprofile::openstack::eqiad1::pdns::auth::service
DNS resolution login.toolforge.orgprofile::openstack::eqiad1::pdns::auth::service
DNS resolution tools-sgegrid-master.tools.SITE1.wikimedia.cloudprofile::openstack::eqiad1::pdns::auth::service
DNS resolution www.wmcloud.orgprofile::openstack::eqiad1::pdns::auth::service
NTP anycast VIPprofile::icinga
PDU-infeed-load-tower-A-phase-Xprofile::icinga
PDU-infeed-load-tower-A-phase-Yprofile::icinga
PDU-infeed-load-tower-A-phase-Zprofile::icinga
PDU-infeed-load-tower-A-single-phaseprofile::icinga
PDU-infeed-load-tower-B-phase-Xprofile::icinga
PDU-infeed-load-tower-B-phase-Yprofile::icinga
PDU-infeed-load-tower-B-phase-Zprofile::icinga
PDU-infeed-load-tower-B-single-phaseprofile::icinga
Planet_content_updatesprofile::icinga
Query_Service_Internal_HTTP_endpointprofile::query_service::wikidata
Query_Service_Local_Blazegraph_endpoint-wcqs-blazegraphprofile::query_service::blazegraph
Query_Service_Local_Blazegraph_endpoint-wdqs-blazegraphprofile::query_service::wikidata
Query_Service_Local_Blazegraph_endpoint-wdqs-categoriesprofile::query_service::categories
Query_Service_Updater_processprofile::query_service::wikidata
Query_Service_Updater_processprofile::query_service::wcqs
Recursive DNS anycast VIPprofile::icinga
WDQS_External_SPARQL_Endpointprofile::query_service::wikidata
_srv_mediawiki-staging_ownedprofile::mediawiki::deployment::server
acme-chief_apiprofile::acme_chief
acme-chief_backendprofile::acme_chief
airflow@analytics_check_dbprofile::airflow
airflow@analytics_check_schedulerprofile::airflow
airflow@analytics_product_check_dbprofile::analytics::cluster::airflow
airflow@analytics_product_check_schedulerprofile::analytics::cluster::airflow
airflow@platform_eng_check_dbprofile::analytics::cluster::airflow
airflow@platform_eng_check_schedulerprofile::analytics::cluster::airflow
airflow@research_check_dbprofile::analytics::cluster::airflow
airflow@research_check_schedulerprofile::analytics::cluster::airflow
airflow@search_check_dbprofile::analytics::cluster::airflow
airflow@search_check_schedulerprofile::analytics::cluster::airflow
airflow@wmde_check_dbprofile::analytics::cluster::airflow
airflow@wmde_check_schedulerprofile::analytics::cluster::airflow
anycast_healthcheckerprofile::dns::auth
anycast_healthcheckerprofile:🐦:anycast_healthchecker_monitoring
anycast_healthcheckerprofile::wmcs::cloud_private_subnet::bgp
aqs_http_rootprofile::aqs
atftpdprofile::installserver::tftp
atlas-ping-SITE-ipv4profile::icinga
atlas-ping-SITE-ipv6profile::icinga
auth dnsprofile::dns::auth
backup_freshnessprofile::backup::director
bacula_directorprofile::backup::director
bacula_sdprofile::backup::storage::common
bacula_sdprofile::backup::storage::main
birdprofile::dns::auth
birdprofile:🐦:anycast
birdprofile::wmcs::cloud_private_subnet::bgp
carbon-cache@ID-stateprofile::graphite::base
carbon-cache_many_createsprofile::graphite::alerts
carbon-cache_overflowprofile::graphite::alerts
carbon-cache_write_errorprofile::graphite::alerts
carbon-frontend-relay-stateprofile::graphite::base
carbon-frontend-relay_dropsprofile::graphite::alerts
carbon-local-relay-stateprofile::graphite::base
carbon-local-relay_dropsprofile::graphite::alerts
cassandra-ID-cqlprofile::cassandra
cassandra-ID-sslprofile::cassandra
cassandra-ID-stateprofile::cassandra
cert_sync_active_nodeprofile::acme_chief
cert_sync_passive_nodeprofile::acme_chief
chartmuseum.service-stateprofile::chartmuseum
check-cinder-snapshot-leaksprofile::openstack::base::cinder
check-cloudinfra-spreadprofile::openstack::eqiad1::keystone::service
check-deployment-prep-spreadprofile::openstack::eqiad1::keystone::service
check-flavor_aggregatesprofile::openstack::eqiad1::nova::fullstack::service
check-flavor_aggregatesprofile::openstack::codfw1dev::nova::fullstack::service
check-neutron-conntrackprofile::openstack::base::neutron::l3_agent
check-tools-spreadprofile::openstack::eqiad1::keystone::service
check-varnish-uds-frontend-/run/varnish-frontend-0.socketprofile::cache::varnish::frontend
check-varnish-uds-frontend-/run/varnish-privileged.socketprofile::cache::varnish::frontend
check_SYSTEMD-UNIT_statusprofile::trafficserver::backendSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::analytics::cluster::airflowSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::piwik::instanceSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::puppetserver::volatileSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::wmcs::backup_glance_imagesSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::wmcs::backup_instancesSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::acme_chiefSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::statographSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::vopsbotSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::puppetmaster::backendSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::pki::multirootcaSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::syslog::centralserverSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::archivaSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::etcd::v3Superseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::openstack::base::cinderSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::netboxSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::wmcs::backup_cinder_volumesSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::airflowSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::analytics::refinery::job::project_namespace_mapSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::kerberos::kdcSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::httpbbSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::kerberos::kadminserverSuperseded by SystemdUnitFailed
check_SYSTEMD-UNIT_statusprofile::puppetmaster::frontendSuperseded by SystemdUnitFailed
check_certificate_expiry_CFSSLCAprofile::pki::multirootca::monitoring
check_chartmuseum_httpprofile::chartmuseum
check_chartmuseum_https_expiryprofile::chartmuseum
check_debian_mirrorprofile::mirrors
check_docker_registry_healthprofile::docker_registry_ha::registry
check_docker_registry_httpsprofile::docker_registry_ha::registry
check_docker_registry_https_expiryprofile::docker_registry_ha::registry
check_dotlsprofile::dns::auth
check_hdfs_topologyprofile::hadoop::master
check_icinga_configprofile::icinga
check_netbox_accountingprofile::netbox
check_netbox_cablesprofile::netbox
check_netbox_coherence_rackprofile::netbox
check_netbox_librenmsprofile::netbox
check_netbox_managementprofile::netbox
check_netbox_networkprofile::netbox
check_netbox_puppetdb_physicalprofile::netbox
check_netbox_puppetdb_virtualprofile::netbox
check_phab_phdprofile::phabricator::monitoring
check_rp_filter_disabledprofile::lvs
check_service_restart_dnsdist.serviceprofile::wikidough
check_service_restart_ntp.serviceprofile::ntp
check_service_restart_pdns-recursor.serviceprofile::wikidough
check_superset_httpprofile::superset
check_systemd_stateprofile::base::production
check_to_check_nagios_pagingprofile::icinga
check_trafficserver_log_fifo_notpurge_backendprofile::trafficserver::backend
check_ubuntu_mirrorprofile::mirrors::ubuntu
check_uncommitted_dns_changesprofile::netbox::automation
check_wikidough_dohprofile::icinga
check_wikidough_doh_ipv4profile::wikidough
check_wikidough_doh_ipv6profile::wikidough
check_wikidough_dotprofile::icinga
check_wikidough_dot_ipv4profile::wikidough
check_wikidough_dot_ipv6profile::wikidough
cirrussearch_SITE_compsuggest_95th_percentilenot-found
cirrussearch_SITE_fulltext_95th_percentilenot-found
cirrussearch_SITE_morelike_95th_percentilenot-found
clamdprofile::vrts
client_bucket_large_fileprofile::base::production
cloudelastic_chi_httpsprofile::icinga
cloudelastic_chi_https_expiryprofile::icinga
cloudelastic_chi_https_publicprofile::icinga
cloudelastic_chi_https_public_expiryprofile::icinga
cloudelastic_omega_httpsprofile::icinga
cloudelastic_omega_https_expiryprofile::icinga
cloudelastic_omega_https_publicprofile::icinga
cloudelastic_omega_https_public_expiryprofile::icinga
cloudelastic_psi_httpsprofile::icinga
cloudelastic_psi_https_expiryprofile::icinga
cloudelastic_psi_https_publicprofile::icinga
cloudelastic_psi_https_public_expiryprofile::icinga
confd-k8s-stateprofile::confd
confd-stateprofile::firewall
confd-stateprofile::cache::base
confd-stateprofile::dns::auth
confd-stateprofile::confd
confd-stateprofile::discovery::client
confd_vcl_reloadprofile::cache::base
conntrack_table_sizeprofile::firewall
conntrack_table_sizeprofile::dns::auth
cpu_microcode_statusprofile::base::production
dbctl_uncommitted_diffsprofile::conftool::dbctl_client
debmonitor-cdn-httpsprofile::debmonitor::server
debmonitor-cdn-https-expiryprofile::debmonitor::server
debmonitor-healthcheckprofile::icinga
debmonitor-healthcheck-expiryprofile::icinga
debmonitor-httpsprofile::debmonitor::server
default_ats_lua_confprofile::trafficserver::backend
disk_spaceprofile::base::production
disk_space_hadoop_workerprofile::hadoop::worker
docker-registry-stateprofile::docker_registry_ha::registry
druid-brokerprofile::druid::broker
druid-coordinatorprofile::druid::coordinator
druid-historicalprofile::druid::historical
druid-middlemanagerprofile::druid::middlemanager
druid-overlordprofile::druid::overlord
dumps_store_load_averageprofile::dumps::generation::server::xmldumps
elasticsearch / cirrus frozen writes - search.svc.SITE.wmnet:PORTprofile::icinga
elasticsearch / masters eligible - search.svc.SITE.wmnet:PORTprofile::icinga
elasticsearch-https-cloudelastic-chi-SITEprofile::elasticsearch::cirrus
elasticsearch-https-cloudelastic-chi-SITE-roprofile::elasticsearch::cirrus
elasticsearch-https-cloudelastic-omega-SITEprofile::elasticsearch::cirrus
elasticsearch-https-cloudelastic-omega-SITE-roprofile::elasticsearch::cirrus
elasticsearch-https-cloudelastic-psi-SITEprofile::elasticsearch::cirrus
elasticsearch-https-cloudelastic-psi-SITE-roprofile::elasticsearch::cirrus
elasticsearch-https-production-search-SITEprofile::elasticsearch::cirrus
elasticsearch-https-production-search-omega-SITEprofile::elasticsearch::cirrus
elasticsearch-https-production-search-psi-SITEprofile::elasticsearch::cirrus
elasticsearch-https-relforge-SITEprofile::elasticsearch::cirrus
elasticsearch-https-relforge-SITE-small-alphaprofile::elasticsearch::cirrus
elasticsearch_setting_check_PORTprofile::elasticsearch::cirrus
elasticsearch_shard_size_check_PORTprofile::elasticsearch::monitor::base_checks
elasticsearch_shard_size_check_search.svc.SITE.wmnet:PORTprofile::icinga
elasticsearch_shards_PORTprofile::elasticsearch::monitor::base_checks
elasticsearch_shards_search.svc.SITE.wmnet:PORTprofile::icinga
elasticsearch_unassigned_shard_check_PORTprofile::elasticsearch::monitor::base_checks
elasticsearch_unassigned_shard_check_search.svc.SITE.wmnet:PORTprofile::icinga
en.m.wp.o-legal-htmlprofile::icinga
en.wb.o-legal-htmlprofile::icinga
en.wp.o-legal-htmlprofile::icinga
endpoints_aqsprofile::aqs
endpoints_kartotherianprofile::maps::apps
ensure_nova_compute_runningprofile::openstack::eqiad1::nova::compute::service
ensure_running_kvm_instancesprofile::openstack::eqiad1::nova::compute::service
ensure_single_nova_compute_procprofile::openstack::eqiad1::nova::compute::service
etcd-stateprofile::etcd::v3
etcd-tlsproxy-sslprofile::etcd::tlsproxy
etcd_cluster_healthprofile::etcd::v3
etcd_mw_configprofile::mediawiki::webserver
etcdmirror-conftool-SITE-wmnet-stateprofile::etcd::replication
eventlogging-jobsprofile::eventlogging::analytics::server
eventstreams_endpointprofile::icinga
ferm_activeprofile::firewall
ferm_activeprofile::dns::auth
freshclamprofile::vrts
ganeti-confdprofile::ganeti
ganeti-mondprofile::ganeti
ganeti-nodedprofile::ganeti
ganeti-wconfdprofile::ganeti
ganeti_memoryprofile::ganeti
gdnsd_checkconfprofile::dns::auth
gdnsd_procprofile::dns::auth
gerritprofile::gerrit
gitlab-httpsprofile::icinga
gitlab-https-expiryprofile::icinga
gitlab-sshprofile::icinga
grafanaprofile::grafana
graphite_apiprofile::graphite::base
graphite_renderprofile::graphite::base
hadoop-hdfs-active-namenodeprofile::hadoop::master
hadoop-hdfs-datanodeprofile::hadoop::worker
hadoop-hdfs-journalnodeprofile::hadoop::worker
hadoop-hdfs-namenodeprofile::hadoop::master
hadoop-hdfs-namenodeprofile::hadoop::master::standby
hadoop-hdfs-zkfcprofile::hadoop::master
hadoop-hdfs-zkfcprofile::hadoop::master::standby
hadoop-mapreduce-historyserverprofile::hadoop::master
hadoop-namenode-backup-ageprofile::hadoop::backup::namenode
hadoop-yarn-nodemanagerprofile::hadoop::worker
hadoop-yarn-resourcemanagerprofile::hadoop::master
haproxyprofile::cache::haproxy
haproxyprofile::mariadb::proxy
haproxyprofile::wmcs::cloudlb::haproxy
haproxy_aliveprofile::mariadb::proxy
haproxy_aliveprofile::wmcs::cloudlb::haproxy
haproxy_failoverprofile::mariadb::proxy::master
haproxy_failoverprofile::mariadb::proxy::multiinstance_replicas
haproxy_https_wikipedia.org_ECDSAprofile::cache::haproxy::monitoring
haproxy_https_wikipedia.org_RSAprofile::cache::haproxy::monitoring
haproxy_https_wikiworkshop.org_ECDSAprofile::cache::haproxy::monitoring
haproxy_https_wikiworkshop.org_RSAprofile::cache::haproxy::monitoring
haproxy_ocsp_freshnessprofile::cache::haproxy::monitoring
haproxy_ocsp_freshness_acme_chiefprofile::cache::haproxy::monitoring
high_iowait_stallingprofile::dumps::distribution::monitoring
hive-metasoreprofile::hive::metastore
hive-server2profile::hive::server
httpprofile::installserver::http
httpprofile::installserver::http_light
httpsprofile::aptrepo::wikimedia
httpsprofile::icinga
httpsprofile::gerrit::proxy
httpsprofile::lists::monitoring
httpsprofile::librenms
https-alerts.wikimedia.org-expiryprofile::idp::client::httpd::site
https-alerts.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-config-master.wikimedia.org-expiryprofile::idp::client::httpd::site
https-config-master.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-gnt-rapi-SITEprofile::ganeti
https-grafana-next-rw.wikimedia.org-expiryprofile::idp::client::httpd::site
https-grafana-next-rw.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-grafana-rw.wikimedia.org-expiryprofile::idp::client::httpd::site
https-grafana-rw.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-graphite.wikimedia.org-expiryprofile::graphite::production
https-graphite.wikimedia.org-unauthorizedprofile::graphite::production
https-hue.wikimedia.org-expiryprofile::hue
https-hue.wikimedia.org-unauthorizedprofile::hue
https-icinga-extmon.wikimedia.org-expiryprofile::icinga::external_monitoring
https-icinga-extmon.wikimedia.org-unauthorizedprofile::icinga::external_monitoring
https-icinga.wikimedia.org-expiryprofile::idp::client::httpd::site
https-icinga.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-klaxon.wikimedia.org-expiryprofile::idp::client::httpd::site
https-klaxon.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-librenms.wikimedia.org-expiryprofile::librenms
https-librenms.wikimedia.org-unauthorizedprofile::librenms
https-orchestrator.wikimedia.org-expiryprofile::orchestrator::web
https-orchestrator.wikimedia.org-unauthorizedprofile::orchestrator::web
https-people.wikimedia.org-expiryprofile::idp::client::httpd::site
https-people.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-piwik.wikimedia.org-expiryprofile::piwik::webserver
https-piwik.wikimedia.org-unauthorizedprofile::piwik::webserver
https-prometheus-SITE.wikimedia.org-expiryprofile::prometheus::web_idp
https-prometheus-SITE.wikimedia.org-unauthorizedprofile::prometheus::web_idp
https-puppetboard.wikimedia.org-expiryprofile::puppetboard
https-puppetboard.wikimedia.org-unauthorizedprofile::puppetboard
https-rt.wikimedia.org-expiryprofile::idp::client::httpd::site
https-rt.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-superset-next.wikimedia.org-expiryprofile::idp::client::httpd::site
https-superset-next.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-superset.wikimedia.org-expiryprofile::idp::client::httpd::site
https-superset.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-turnilo.wikimedia.org-expiryprofile::idp::client::httpd::site
https-turnilo.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https-yarn.wikimedia.org-expiryprofile::idp::client::httpd::site
https-yarn.wikimedia.org-unauthorizedprofile::idp::client::httpd::site
https_archivaprofile::archiva::proxy
https_blogprofile::icinga
https_ncredir_non-canonical-redirect-IDprofile::ncredir
https_planetprofile::icinga
https_policyprofile::icinga
https_status-wikimediaprofile::icinga
https_wikitech-staticprofile::icinga
https_wmfusercontentprofile::icinga
hue-gunicornprofile::hue
hue-kt-renewerprofile::hue
ircdprofile::mw_rc_irc
ircechoprofile::icinga::ircbot
ircecho-processprofile::mw_rc_irc
jenkinsprofile::releases::mediawiki
jenkinsprofile::ci::jenkins
kafkaprofile::kafka::broker
kafka-broker-tlsprofile::kafka::broker
kafka-mirror-main-SITE_to_jumbo-SITE-consume_rateprofile::prometheus::alerts
kafka-mirror-main-SITE_to_jumbo-SITE-consumer_max_lagprofile::prometheus::alerts
kafka-mirror-main-SITE_to_jumbo-SITE-dropped_messagesprofile::prometheus::alerts
kafka-mirror-main-SITE_to_jumbo-SITE-produce_rateprofile::prometheus::alerts
kafka-mirror-main-SITE_to_jumbo-SITE@0profile::kafka::mirror
kafka-mirror-main-SITE_to_main-SITE-consume_rateprofile::prometheus::alerts
kafka-mirror-main-SITE_to_main-SITE-consumer_max_lagprofile::prometheus::alerts
kafka-mirror-main-SITE_to_main-SITE-dropped_messagesprofile::prometheus::alerts
kafka-mirror-main-SITE_to_main-SITE-produce_rateprofile::prometheus::alerts
kafka-mirror-main-SITE_to_main-SITE@0profile::kafka::mirror
karapaceprofile::karapace::main
krb-kadmin-serverprofile::kerberos::kadminserver
krb-kdcprofile::kerberos::kdc
krb-kpropdprofile::kerberos::kadminserver
labs_private_mergedprofile::puppetmaster::frontend
ldapprofile::openldap_clouddev
ldapprofile::openldap
librenmsprofile::librenms
librenms-ssl-expiryprofile::librenms
load_averageprofile::swift::storage
load_averageprofile::thanos::swift::backend
mailman3_queueprofile::lists::monitoring
mailman3_runnersprofile::lists::monitoring
mailman_archivesprofile::lists::monitoring
mailman_hours_until_empty_outbound_queueprofile::lists::monitoring
mailman_listinfoprofile::lists::monitoring
mailman_listinfo_ssl_expiryprofile::lists::monitoring
mailman_queueprofile::lists::monitoring
mariadb-prolonged-lag_SHARDprofile::mariadb::replication_lag
mariadb_disk_spaceprofile::mariadb::parsercache
mariadb_disk_spaceprofile::mariadb::core
mariadb_disk_spaceprofile::wmcs::db::wikireplicas::mariadb_multiinstance
mariadb_disk_spaceprofile::mariadb::misc
mariadb_disk_spaceprofile::mariadb::monitor
mariadb_disk_spaceprofile::openstack::base::pdns::auth::db
mariadb_disk_spaceprofile::mariadb::core::multiinstance
mariadb_disk_spaceprofile::openstack::codfw1dev::pdns::auth::db
mariadb_disk_spaceprofile::mariadb::dbstore_multiinstance
mariadb_disk_spaceprofile::mariadb::sanitarium_multiinstance
mariadb_disk_spaceprofile::mariadb::misc::db_inventory
mariadb_disk_spacenot-found
mariadb_disk_spaceprofile::mariadb::misc::analytics::multiinstance
mariadb_disk_spaceprofile::mariadb::misc::multiinstance
mariadb_dump_SHARD_SITEprofile::dbbackups::check
mariadb_dump_analytics_meta_SITEprofile::dbbackups::check
mariadb_dump_backup1-SITE_SITEprofile::dbbackups::check
mariadb_dump_db_inventory_SITEprofile::dbbackups::check
mariadb_dump_matomo_SITEprofile::dbbackups::check
mariadb_event_scheduler_SHARDprofile::mariadb::parsercache
mariadb_event_scheduler_test_SHARDprofile::mariadb::core_test
mariadb_memoryprofile::mariadb::parsercache
mariadb_memoryprofile::mariadb::core
mariadb_memoryprofile::wmcs::db::wikireplicas::mariadb_multiinstance
mariadb_memoryprofile::mariadb::misc
mariadb_memoryprofile::mariadb::core_test
mariadb_memoryprofile::mariadb::core::multiinstance
mariadb_memoryprofile::mariadb::dbstore_multiinstance
mariadb_memoryprofile::mariadb::sanitarium_multiinstance
mariadb_memoryprofile::mariadb::misc::db_inventory
mariadb_memorynot-found
mariadb_memoryprofile::mariadb::misc::analytics::multiinstance
mariadb_read_only_SHARDprofile::mariadb::parsercache
mariadb_read_only_SHARDprofile::mariadb::core
mariadb_read_only_SHARDprofile::wmcs::db::wikireplicas::mariadb_multiinstance
mariadb_read_only_SHARDprofile::mariadb::misc
mariadb_read_only_SHARDprofile::mariadb::core::multiinstance
mariadb_read_only_SHARDprofile::mariadb::dbstore_multiinstance
mariadb_read_only_SHARDprofile::mariadb::sanitarium_multiinstance
mariadb_read_only_SHARDnot-found
mariadb_read_only_SHARDprofile::mariadb::misc::multiinstance
mariadb_read_only_analytics_metaprofile::mariadb::misc::analytics::multiinstance
mariadb_read_only_backup1-SITEprofile::mariadb::misc
mariadb_read_only_db_inventoryprofile::mariadb::misc::db_inventory
mariadb_read_only_matomoprofile::mariadb::misc::analytics::multiinstance
mariadb_read_only_stagingprofile::mariadb::dbstore_multiinstance
mariadb_read_only_test_SHARDprofile::mariadb::core_test
mariadb_read_only_wikireplica_SHARDprofile::wmcs::db::wikireplicas::mariadb_multiinstance
mariadb_replica_io_state_SHARDprofile::mariadb::parsercache
mariadb_replica_io_state_SHARDprofile::mariadb::core
mariadb_replica_io_state_SHARDprofile::wmcs::db::wikireplicas::mariadb_multiinstance
mariadb_replica_io_state_SHARDprofile::mariadb::misc
mariadb_replica_io_state_SHARDprofile::mariadb::core::multiinstance
mariadb_replica_io_state_SHARDprofile::mariadb::dbstore_multiinstance
mariadb_replica_io_state_SHARDprofile::mariadb::sanitarium_multiinstance
mariadb_replica_io_state_SHARDnot-found
mariadb_replica_io_state_SHARDprofile::mariadb::misc::multiinstance
mariadb_replica_io_state_analytics-meta-replicaprofile::analytics::database::meta
mariadb_replica_io_state_analytics_metaprofile::mariadb::misc::analytics::multiinstance
mariadb_replica_io_state_backup1-SITEprofile::mariadb::misc
mariadb_replica_io_state_db_inventoryprofile::mariadb::misc::db_inventory
mariadb_replica_io_state_matomoprofile::mariadb::misc::analytics::multiinstance
mariadb_replica_io_state_stagingprofile::mariadb::dbstore_multiinstance
mariadb_replica_io_state_test_SHARDprofile::mariadb::core_test
mariadb_replica_sql_lag_SHARDprofile::mariadb::parsercache
mariadb_replica_sql_lag_SHARDprofile::mariadb::core
mariadb_replica_sql_lag_SHARDprofile::wmcs::db::wikireplicas::mariadb_multiinstance
mariadb_replica_sql_lag_SHARDprofile::mariadb::misc
mariadb_replica_sql_lag_SHARDprofile::mariadb::core::multiinstance
mariadb_replica_sql_lag_SHARDprofile::mariadb::dbstore_multiinstance
mariadb_replica_sql_lag_SHARDprofile::mariadb::sanitarium_multiinstance
mariadb_replica_sql_lag_SHARDnot-found
mariadb_replica_sql_lag_SHARDprofile::mariadb::misc::multiinstance
mariadb_replica_sql_lag_analytics-meta-replicaprofile::analytics::database::meta
mariadb_replica_sql_lag_analytics_metaprofile::mariadb::misc::analytics::multiinstance
mariadb_replica_sql_lag_backup1-SITEprofile::mariadb::misc
mariadb_replica_sql_lag_db_inventoryprofile::mariadb::misc::db_inventory
mariadb_replica_sql_lag_matomoprofile::mariadb::misc::analytics::multiinstance
mariadb_replica_sql_lag_stagingprofile::mariadb::dbstore_multiinstance
mariadb_replica_sql_lag_test_SHARDprofile::mariadb::core_test
mariadb_replica_sql_state_SHARDprofile::mariadb::parsercache
mariadb_replica_sql_state_SHARDprofile::mariadb::core
mariadb_replica_sql_state_SHARDprofile::wmcs::db::wikireplicas::mariadb_multiinstance
mariadb_replica_sql_state_SHARDprofile::mariadb::misc
mariadb_replica_sql_state_SHARDprofile::mariadb::core::multiinstance
mariadb_replica_sql_state_SHARDprofile::mariadb::dbstore_multiinstance
mariadb_replica_sql_state_SHARDprofile::mariadb::sanitarium_multiinstance
mariadb_replica_sql_state_SHARDnot-found
mariadb_replica_sql_state_SHARDprofile::mariadb::misc::multiinstance
mariadb_replica_sql_state_analytics-meta-replicaprofile::analytics::database::meta
mariadb_replica_sql_state_analytics_metaprofile::mariadb::misc::analytics::multiinstance
mariadb_replica_sql_state_backup1-SITEprofile::mariadb::misc
mariadb_replica_sql_state_db_inventoryprofile::mariadb::misc::db_inventory
mariadb_replica_sql_state_matomoprofile::mariadb::misc::analytics::multiinstance
mariadb_replica_sql_state_stagingprofile::mariadb::dbstore_multiinstance
mariadb_replica_sql_state_test_SHARDprofile::mariadb::core_test
mariadb_snapshot_SHARD_SITEprofile::dbbackups::check
mediawiki-installation DSH groupprofile::mediawiki::common
mediawiki-installation DSH groupprofile::openstack::base::wikitech::web
mediawiki-installation DSH groupprofile::dumps::generation::worker::common
mediawiki_accountcreation_errorsprofile::graphite::alerts
mediawiki_bad_tokenprofile::graphite::alerts
mediawiki_centralauth_errorsprofile::graphite::alerts
mediawiki_cirrus_pool_counter_rejections_rateprofile::icinga
mediawiki_cirrus_update_rate_SITEprofile::icinga
mediawiki_cirrussearch_indices_high_fix_rateprofile::icinga
mediawiki_config_mergedprofile::mediawiki::deployment::server
mediawiki_session_lossprofile::graphite::alerts
memcachedprofile::mediawiki::mcrouter_wancache
memcachedprofile::openstack::base::cloudweb_mcrouter
memcachedprofile::memcached::instance
memcachedprofile::idp
memcachedprofile::thanos::swift::frontend
memcachedprofile::openstack::eqiad1::designate::service
memcachedprofile::swift::proxy
memcachedprofile::openstack::codfw1dev::designate::service
memcachedprofile::openstack::eqiad1::cloudweb_mcrouter
memcached_socketprofile::mediawiki::mcrouter_wancache
minio_serverprofile::mediabackup::storage
mw_wikiversion_differenceprofile::mediawiki::monitor_versions
mw_wikiversion_differenceprofile::openstack::base::wikitech::web
mw_wikiversion_differenceprofile::dumps::generation::worker::common
mw_wikiversion_differenceprofile::mediawiki::deployment::server
mysql_analytics-metaprofile::analytics::database::meta
mysqldprofile::mariadb::parsercache
mysqldprofile::mariadb::core
mysqldprofile::wmcs::db::wikireplicas::mariadb_multiinstance
mysqldprofile::mariadb::misc
mysqldprofile::mariadb::monitor
mysqldprofile::openstack::base::pdns::auth::db
mysqldprofile::mariadb::core::multiinstance
mysqldprofile::openstack::codfw1dev::pdns::auth::db
mysqldprofile::mariadb::dbstore_multiinstance
mysqldprofile::mariadb::sanitarium_multiinstance
mysqldprofile::mariadb::misc::db_inventory
mysqldnot-found
mysqldprofile::mariadb::misc::analytics::multiinstance
mysqldprofile::mariadb::misc::multiinstance
network_in_saturatedprofile::dumps::distribution::monitoring
network_out_saturatedprofile::dumps::distribution::monitoring
nfsprofile::dumps::distribution::nfs
nfsprofile::dumps::nfs
nfsdprofile::dumps::nfs
ns0-v4profile::dns::auth::monitoring::global
ns1-v4profile::dns::auth::monitoring::global
ns2-v4profile::dns::auth::monitoring::global
ntp peersprofile::ntp
nutcrackerprofile::openstack::codfw1dev::nutcracker
nutcrackerprofile::openstack::base::nutcracker
nutcracker_portprofile::openstack::codfw1dev::nutcracker
nutcracker_portprofile::openstack::base::nutcracker
opcacheprofile::mediawiki::php::monitoring
opcacheprofile::parsoid
opensearch_shard_size_check_PORTprofile::opensearch::monitoring::base_checks
opensearch_shards_PORTprofile::opensearch::monitoring::base_checks
opensearch_unassigned_shard_check_PORTprofile::opensearch::monitoring::base_checks
orchestrator_processprofile::orchestrator::monitoring
orchestrator_tcp_portprofile::orchestrator::monitoring
package_builder_rsyncprofile::package_builder
php7.4-fpm-stateprofile::mediawiki::php::monitoring
php7.4-fpm-stateprofile::parsoid
poolcounterdprofile::poolcounter
poolcounterd_port_PORTprofile::poolcounter
postgres-rep-lagprofile::maps::osm_replica
postgres-rep-lagprofile::puppetdb::database
postgres-rep-lagprofile::netbox::db
presto-serverprofile::presto::server
procs_mailman3profile::lists::monitoring
procs_mailman3_webprofile::lists::monitoring
pt-heartbeat-wikimedia-stateprofile::mariadb::parsercache
pt-heartbeat-wikimedia-stateprofile::mariadb::core
pt-heartbeat-wikimedia-stateprofile::mariadb::misc
pt-heartbeat-wikimedia-stateprofile::mariadb::core_test
pt-heartbeat-wikimedia-stateprofile::mariadb::misc::db_inventory
pt-heartbeat-wikimedia-statenot-found
puppet_mergedprofile::puppetmaster::backend
puppet_mergedprofile::puppetmaster::frontend
puppetmaster_backend_httpsprofile::puppetmaster::backend
puppetmaster_backend_httpsprofile::puppetmaster::frontend
puppetmaster_httpsprofile::puppetmaster::frontend
purged-stateprofile::cache::purge
pybalprofile::pybal
pybal_backendsprofile::pybal
pybal_etcd_connectionsprofile::pybal
pybal_ipvs_diffprofile::pybal
raid_mdprofile::base::production
raid_megaraidprofile::base::production
raid_perc_raidprofile::base::production
raid_ssacliprofile::base::production
recursive dns IPV4profile::dns::recursor
recursive dns IPV6profile::dns::recursor
redis_status_on_port_PORTprofile::redis::slave
redis_status_on_port_PORTprofile::redis::master
restbase_http_rootprofile::restbase
restbase_instance_spaceprofile::restbase
routinator-processprofile::rpkivalidator
rpkivalidator-rtr-monprofile::rpkivalidator
samplicator-processprofile::samplicator
search_backend_failure_countnot-found
smtpprofile::mail::mx
smtpprofile::lists::monitoring
spamdprofile::vrts
spamdprofile::mail::mx
spamdprofile::lists
squidprofile::installserver::proxy
sshprofile::base::production
statsd-proxyprofile::statsd
statsvprofile::webperf::processors
supersetprofile::superset
swift-https-backendprofile::swift::proxy
swift-https-frontendprofile::swift::proxy
syslog::centralserver centrallogPORT syslog-tlsprofile::syslog::centralserver
thanos-swift-httpsprofile::thanos::swift::frontend
timesynd_ntp_statusprofile::base::production
tools-checker-dumpsprofile::icinga
tools-checker-etcd-k8sprofile::icinga
tools-checker-grid-continuous-busterprofile::icinga
tools-checker-grid-start-busterprofile::icinga
tools-checker-k8s-node-readyprofile::icinga
tools-checker-labs-dns-privateprofile::icinga
tools-checker-ldapprofile::icinga
tools-checker-nfs-homeprofile::icinga
tools-checker-redisprofile::icinga
tools-checker-selfprofile::icinga
tools-checker-toolscronprofile::icinga
traffic-pool-stateprofile::cache::base
traffic_manager_backendprofile::trafficserver::backend
traffic_manager_backend_check_httpprofile::trafficserver::backend
traffic_server_backendprofile::trafficserver::backend
trafficserver_backend_exporter_check_httpprofile::trafficserver::backend
trafficserver_exporter_backendprofile::trafficserver::backend
trafficserver_skipped_logsprofile::trafficserver::backend
turniloprofile::druid::turnilo
update-etcd-mw-config-lastindex-stateprofile::icinga
url_downloaderprofile::url_downloader
uwsgi-acme-chiefprofile::acme_chief
uwsgi-bituprofile::idm
uwsgi-debmonitorprofile::debmonitor::server
uwsgi-graphite-webprofile::graphite::base
uwsgi-netboxprofile::netbox
uwsgi-netbox-scriptproxyprofile::netbox::scripts
uwsgi-nrpe-puppetboardprofile::puppetboard
uwsgi-puppetboardprofile::puppetboard
uwsgi-puppetdb-microserviceprofile::puppetdb
varnishkafka-eventloggingprofile::cache::kafka::eventlogging
varnishkafka-statsvprofile::cache::kafka::statsv
varnishkafka-webrequestprofile::cache::base
wcqs-blazegraph-_processprofile::query_service::blazegraph
wdqs-blazegraph-_processprofile::query_service::wikidata
wdqs-categories-_processprofile::query_service::categories
wikitech-static-main-pageprofile::icinga
wikitech-static-syncprofile::icinga
wikitech-static-versionprofile::icinga
zookeeperprofile::zookeeper::server
zookeeper_client_connsprofile::zookeeper::server
zuulprofile::zuul::server
zuul_gearmanprofile::zuul::server
zuul_gearman_wait_queueprofile::zuul::server
zuul_mergerprofile::zuul::merger
zuul_merger_git_daemonprofile::zuul::merger

Change 975284 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] icinga: add alert audit via puppetdb

https://gerrit.wikimedia.org/r/975284

Change 975284 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: add alert audit via puppetdb

https://gerrit.wikimedia.org/r/975284

fgiunchedi claimed this task.

I'm boldly resolving this: the progress indicator is at this tab https://docs.google.com/spreadsheets/d/19nxCXldb804TJCXGy4Z2BHG_1wRksRnKcPC6sXfjQuM/edit#gid=701141702 namely the "TODO" pivot table that lists all icinga checks we have yet to migrate