Page MenuHomePhabricator
Create Task
Maniphest T350694

Infrastructure Foundation Alerts to migrate
Open, MediumPublicGoal
Actions

Description

The following are alerts for Infrastructure Foundation to migrate, sorted by number of devices with the alert defined:

  • ssh
  • disk_space
  • cpu_microcode_status
  • client_bucket_large_file - Talk to @jhathaway and this alert just goes away.
  • timesynd_ntp_status
  • check_systemd_state
  • ferm_active - We're replacing Ferm with nftables, which is alerting via Alertmanager. Not migrating, but deprecating.
  • conntrack_table_size
  • raid_md
  • raid_megaraid
  • raid_perc_raid
  • ganeti_memory
  • ganeti-noded
  • ganeti-mond
  • ganeti-confd
  • load_average
  • raid_ssacli - This will become irrelevant when we decommission the final HP servers. I don't believe writing an exporter for this is a good use of time.
  • ntp
  • ganeti-wconfd <- This cannot be removed due to the current implementation of services in Ganeti. Each service is wrapped using supervisord, preventing us from acting on a specific service, and we need to verify that only one wconfd is running.
  • squid
  • samplicator-process
  • puppetmaster_backend_https
  • puppet_merged
  • url_downloader
  • check_netbox_ganeti_SITE_sync_status
  • uwsgi-netbox-scriptproxy
  • uwsgi-netbox
  • uwsgi-debmonitor - This goes away with the migration to Debian packages. After this it's a standard systemd monitoring
  • debmonitor-https
  • debmonitor-cdn-https-expiry
  • debmonitor-cdn-https
  • uwsgi-puppetdb-microservice
  • uwsgi-puppetboard
  • uwsgi-nrpe-puppetboard
  • puppetmaster_https
  • krb-kdc
  • check_netbox_housekeeping_status
  • check_netbox_ganeti_SITE02_sync_status
  • check_netbox_ganeti_SITE01_sync_status
  • check_cfssl-multirootca_status
  • check_certificate_expiry_syslog
  • check_certificate_expiry_puppet_rsa
  • check_certificate_expiry_network_devices
  • check_certificate_expiry_discovery - Unable to locate this one.
  • check_certificate_expiry_debmonitor
  • check_certificate_expiry_cloud_wmnet_ca
  • uwsgi-bitu
  • package_builder_rsync
  • krb-kpropd
  • krb-kadmin-server
  • hue-gunicorn
  • check_uncommitted_dns_changes
  • check_netbox_puppetdb_virtual
  • check_netbox_puppetdb_physical
  • check_netbox_network
  • check_netbox_management
  • check_netbox_librenms
  • check_netbox_ganeti_SITE_test_sync_status
  • check_netbox_coherence_rack
  • check_netbox_cables
  • check_netbox_accounting
  • check_debian_mirror
  • http and squid checks for installserver
  • http checks for aptrepo
  • smtp checks for mail servers
  • ldap checks

Original list from o11y: https://phabricator.wikimedia.org/P53115

Migrated?(Y/N)TitleResource TypeCommandFileProfiles
Ncheck_certificate_expiry_XNrpe::Monitor_service/usr/bin/openssl/srv/puppet_code/environments/production/modules/profile/manifests/pki/multirootca/monitoring.pp:18profile::pki::multirootca, profile::pki::multirootca::monitoring
Npackage_builder_rsyncMonitoring::Servicecheck_tcp/srv/puppet_code/environments/production/modules/profile/manifests/package_builder.pp:25profile::package_builder
Nferm_activeNrpe::Monitor_service/usr/local/lib/nagios/plugins/check_ferm/srv/puppet_code/environments/production/modules/profile/manifests/firewall.pp:145profile::firewall, profile::backup::director, profile::backup::storage::main, profile::backup::storage::common, profile::dns::auth::monitoring, profile::dns::auth, profile::ci::firewall, profile::mariadb::proxy
Ndebmonitor-healthcheck-expiryMonitoring::Servicecheck_https_expiry/srv/puppet_code/environments/production/modules/icinga/manifests/monitor/debmonitor.pp:10profile::icinga
Ncheck_cfssl-multirootca_statusNrpe::Monitor_service/usr/local/lib/nagios/plugins/check_systemd_unit_status/srv/puppet_code/environments/production/modules/systemd/manifests/monitor.pp:23profile::pki::multirootca
NldapMonitoring::Servicecheck_ldap/srv/puppet_code/environments/production/modules/profile/manifests/openldap_clouddev.pp:56profile::openldap_clouddev
Nhttps-gnt-rapi-XNrpe::Monitor_service/usr/lib/nagios/plugins/check_http/srv/puppet_code/environments/production/modules/profile/manifests/ganeti.pp:230profile::ganeti
Ncheck_uncommitted_dns_changesNrpe::Monitor_service/usr/lib/nagios/plugins/check_json_file/srv/puppet_code/environments/production/modules/profile/manifests/netbox/automation.pp:92profile::netbox::automation
Ycpu_microcode_statusNrpe::Monitor_service/usr/local/lib/nagios/plugins/check_microcode/srv/puppet_code/environments/production/modules/base/manifests/kernel.pp:100profile::base, profile::base::production
NsquidMonitoring::Servicecheck_tcp/srv/puppet_code/environments/production/modules/profile/manifests/installserver/proxy.pp:57profile::installserver::proxy
Nraid_mdNrpe::Monitor_service/usr/local/lib/nagios/plugins/check_raid/srv/puppet_code/environments/production/modules/raid/manifests/md.pp:51profile::base::production, profile::monitoring
Ndebmonitor-httpsMonitoring::Servicecheck_https_client_auth_puppet/srv/puppet_code/environments/production/modules/profile/manifests/debmonitor/server.pp:188profile::debmonitor::server
NldapMonitoring::Servicecheck_ldap/srv/puppet_code/environments/production/modules/profile/manifests/openldap.pp:60profile::openldap
NhttpMonitoring::Servicecheck_http/srv/puppet_code/environments/production/modules/profile/manifests/installserver/http.pp:18profile::installserver::http
Ndebmonitor-cdn-https-expiryMonitoring::Servicecheck_https_expiry/srv/puppet_code/environments/production/modules/profile/manifests/debmonitor/server.pp:181profile::debmonitor::server
NsmtpMonitoring::Servicecheck_smtp_tls_le/srv/puppet_code/environments/production/modules/profile/manifests/postfix/mx.pp:352profile::community_civicrm::mail, profile::community_civicrm, profile::postfix::mx
Nraid_broadcom_raidNrpe::Monitor_service/usr/local/lib/nagios/plugins/get-raid-status-broadcom/srv/puppet_code/environments/production/modules/raid/manifests/broadcom.pp:21profile::base::production, profile::monitoring
Ndebmonitor-healthcheckMonitoring::Servicecheck_https_redirect/srv/puppet_code/environments/production/modules/icinga/manifests/monitor/debmonitor.pp:10profile::icinga
Ndbctl_uncommitted_diffsNrpe::Monitor_service/usr/local/lib/nagios/plugins/check_dbctl_uncommitted_diffs/srv/puppet_code/environments/production/modules/profile/manifests/conftool/dbctl_client.pp:12profile::conftool::dbctl_client
Ndebmonitor-cdn-httpsMonitoring::Servicecheck_https_redirect/srv/puppet_code/environments/production/modules/profile/manifests/debmonitor/server.pp:174profile::debmonitor::server
Nraid_megaraidNrpe::Monitor_service/usr/local/lib/nagios/plugins/check_raid/srv/puppet_code/environments/production/modules/raid/manifests/megaraid.pp:17profile::base::production, profile::monitoring
NhttpMonitoring::Servicecheck_http/srv/puppet_code/environments/production/modules/profile/manifests/installserver/http_light.pp:31profile::installserver::http_light
NhttpsMonitoring::Servicecheck_ssl_http_letsencrypt/srv/puppet_code/environments/production/modules/profile/manifests/aptrepo/wikimedia.pp:103profile::aptrepo::wikimedia

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Remove obsolete Icinga checkMuehlenhoffoperations/puppetproduction+0 -136
P:ganeti: Absent checks for generic Ganeti services.Slyngshedeoperations/puppetproduction+3 -0
P:debmonitor::server Add CDN endpoint check.Slyngshedeoperations/puppetproduction+7 -0
netbox: enable nrpe2nodexp wrapper on check_uncommitted_dns_changes checkTiziano Foglioperations/puppetproduction+9 -8
dbctl: enable nrpe2nodexp wrapper on dbctl_uncommitted_diffsTiziano Foglioperations/puppetproduction+11 -10
base: remove check_microcodeTiziano Foglioperations/puppetproduction+0 -90
Netbox: simplify querySlyngshedeoperations/alertsmaster+8 -18
Netbox alerting: Add remaining netbox alert.Slyngshedeoperations/alertsmaster+110 -0
P:systemd::timesyncd remove deprecated Icinga checkSlyngshedeoperations/puppetproduction+1 -10
Alert when mirrors become out of dateSlyngshedeoperations/alertsmaster+78 -0
P:firewall remove unused codeSlyngshedeoperations/puppetproduction+0 -5
P:mirrors add file age exporterSlyngshedeoperations/puppetproduction+7 -0
P:firewall remove connection tracking monitoringSlyngshedeoperations/puppetproduction+1 -8
P:firewall absent conntrack_table_size monitoring.Slyngshedeoperations/puppetproduction+1 -0
P:firewall absent conntrack_table_size monitoring.Slyngshedeoperations/puppetproduction+1 -0
sre: limit netbox reports alerts to eqiad and codfwFilippo Giunchedioperations/alertsmaster+1 -0
P:systemd::timesyncd absent monitoring, handled by AlertManagerSlyngshedeoperations/puppetproduction+1 -1
base: absent check_microcodeFilippo Giunchedioperations/puppetproduction+2 -1
Netbox: Update runbook, add dashboard and physicalhosts report.Slyngshedeoperations/alertsmaster+28 -5
Start migrating Netbox alerts from Icinga.Slyngshedeoperations/alertsmaster+37 -0
P:ganeti Prometheus monitoring of ganeti noded services.Slyngshedeoperations/puppetproduction+8 -0
P:hue absent Icinga monitoring.Slyngshedeoperations/puppetproduction+4 -0
PKI: Switch alerts to use the x509 metric.Slyngshedeoperations/alertsmaster+25 -23
Load average for Swift clusterSlyngshedeoperations/alertsmaster+49 -0
Puppetmaster: Adjust alerting interval.Slyngshedeoperations/alertsmaster+1 -1
C:prometheus::node_exporter allow CPU flags collectionSlyngshedeoperations/puppetproduction+7 -0
C:ganeti::prometheus Collect process informationSlyngshedeoperations/puppetproduction+96 -0
P:pki::multirootca::monitoring Collect metrics from intermediate.Slyngshedeoperations/puppetproduction+8 -0
P:kerberos::kdc absent Icinga checkSlyngshedeoperations/puppetproduction+1 -0
P:kerberos::replication absent Icinga checkSlyngshedeoperations/puppetproduction+1 -0
C:puppetmaster::monitoring Disable Icinga merge check.Slyngshedeoperations/puppetproduction+9 -3
Puppetmaster: Alert when unmerged changes exists in Puppet repo.Slyngshedeoperations/alertsmaster+46 -0
P:puppetboard absent Icinga checks for PuppetBoard.Slyngshedeoperations/puppetproduction+1 -0
P:netbox monitoring not required on systemd timer.Slyngshedeoperations/puppetproduction+13 -15
P:puppetdb::microservice absent uwsgi Icinga check.Slyngshedeoperations/puppetproduction+3 -2
D:service::uwsgi Parse the Icinga disable flag to uwsgi:app.Slyngshedeoperations/puppetproduction+1 -0
P:ganeti: Remove Icinga memory monitoring.Slyngshedeoperations/puppetproduction+1 -0
C:puppetmaster::monitoring Absent Icinga monitoringSlyngshedeoperations/puppetproduction+2 -0
Monitoring of PKI infrastructure certs.Slyngshedeoperations/alertsmaster+69 -0
P:puppet::client_bucket remove monitoring.Slyngshedeoperations/puppetproduction+1 -1
P:kerberos::kadminserver absent Icinga checkSlyngshedeoperations/puppetproduction+1 -0
P:puppet::client_bucket Start moving monitoring to PrometheusSlyngshedeoperations/puppetproduction+97 -0
P::installserver::proxy Blackbox monitoringSlyngshedeoperations/puppetproduction+6 -0
D:uwsgi::app Allow disabling of monitoring.Slyngshedeoperations/puppetproduction+4 -2
D:prometheus::blackbox::check::tcp allow specifying runbook.Slyngshedeoperations/puppetproduction+9 -2
P:url_downloader decommission Icinga checkSlyngshedeoperations/puppetproduction+2 -0
C:samplicator Icinga monitoring is not required.Slyngshedeoperations/puppetproduction+1 -0
P:url_downloader absent Icinga check.Slyngshedeoperations/puppetproduction+2 -0
P:debmonitor::server rework debmonitor http monitoring.Slyngshedeoperations/puppetproduction+8 -4
Netfilter: Remove exclude filter.Slyngshedeoperations/alertsmaster+2 -2
Ganeti memory pressure alerting.Slyngshedeoperations/alertsmaster+66 -0
Netfilter max connection tracking entires.Slyngshedeoperations/alertsmaster+71 -0
P:debmonitor::server check host for CDN.Slyngshedeoperations/puppetproduction+1 -0
C:puppetmaster::monitoring labs/private uses master branch.Slyngshedeoperations/puppetproduction+1 -1
P:debmonitor::server Add Prometheus Blackbox monitoringSlyngshedeoperations/puppetproduction+17 -0
C:puppetmaster::monitoring missing execute bit on script.Slyngshedeoperations/puppetproduction+1 -0
C:puppetmaster::monitoring Reenable Prometheus data collection.Slyngshedeoperations/puppetproduction+6 -5
Add warning for OOM killer.Slyngshedeoperations/alertsmaster+45 -0
C:puppetmaster::monitoring disable timersSlyngshedeoperations/puppetproduction+2 -2
C:puppetmaster::monitoring Prometheus stats for Puppetmerge.Slyngshedeoperations/puppetproduction+113 -0
C:puppetmaster::monitoring Blackbox monitoringSlyngshedeoperations/puppetproduction+19 -0
C:puppetmaster::monitoring remove blackbox checkSlyngshedeoperations/puppetproduction+0 -16
C:puppetmaster::monitoring use FQDN for blackboxSlyngshedeoperations/puppetproduction+2 -3
C:puppetmaster::monitoring Add Prometheus monitoringSlyngshedeoperations/puppetproduction+17 -0
Blackbox alerting for urldownloadersSlyngshedeoperations/alertsmaster+45 -0
P:url_downloader add blackbox exporter.Slyngshedeoperations/puppetproduction+5 -0
Alert on degraded MD RAID devices.Slyngshedeoperations/alertsmaster+82 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Feb 14 2024, 1:13 PM

Change 1003413 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:netbox monitoring not required on systemd timer.

https://gerrit.wikimedia.org/r/1003413

gerritbot added a comment.Feb 15 2024, 2:07 PM

Change 1003761 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/alerts@master] Puppetmaster: Alert when unmerged changes exists in Puppet repo.

https://gerrit.wikimedia.org/r/1003761

gerritbot added a comment.Feb 16 2024, 7:52 AM

Change 1003405 merged by Slyngshede:

[operations/puppet@production] P:puppetdb::microservice absent uwsgi Icinga check.

https://gerrit.wikimedia.org/r/1003405

SLyngshede-WMF updated the task description. (Show Details)Feb 16 2024, 7:52 AM
gerritbot added a comment.Feb 16 2024, 7:54 AM

Change 1003413 merged by Slyngshede:

[operations/puppet@production] P:netbox monitoring not required on systemd timer.

https://gerrit.wikimedia.org/r/1003413

SLyngshede-WMF updated the task description. (Show Details)Feb 16 2024, 7:55 AM
gerritbot added a comment.Feb 16 2024, 8:38 AM

Change 1003406 merged by Slyngshede:

[operations/puppet@production] P:puppetboard absent Icinga checks for PuppetBoard.

https://gerrit.wikimedia.org/r/1003406

SLyngshede-WMF updated the task description. (Show Details)Feb 16 2024, 8:39 AM
gerritbot added a comment.Feb 16 2024, 11:01 AM

Change 1004061 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] C:ganeti::prometheus Collect process information

https://gerrit.wikimedia.org/r/1004061

gerritbot added a comment.Feb 19 2024, 9:59 AM

Change 1004619 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/alerts@master] Load average for Swift cluster

https://gerrit.wikimedia.org/r/1004619

gerritbot added a comment.Feb 20 2024, 1:12 PM

Change 1003761 merged by jenkins-bot:

[operations/alerts@master] Puppetmaster: Alert when unmerged changes exists in Puppet repo.

https://gerrit.wikimedia.org/r/1003761

gerritbot added a comment.Feb 22 2024, 12:17 PM

Change 1005736 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/alerts@master] Puppetmaster: Adjust alerting interval.

https://gerrit.wikimedia.org/r/1005736

gerritbot added a comment.Feb 22 2024, 12:24 PM

Change 1005743 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] C:puppetmaster::monitoring Disable Icinga merge check.

https://gerrit.wikimedia.org/r/1005743

gerritbot added a comment.Feb 26 2024, 7:37 AM

Change 1005743 merged by Slyngshede:

[operations/puppet@production] C:puppetmaster::monitoring Disable Icinga merge check.

https://gerrit.wikimedia.org/r/1005743

SLyngshede-WMF updated the task description. (Show Details)Feb 26 2024, 7:38 AM
gerritbot added a comment.Feb 27 2024, 8:14 AM

Change 1003383 merged by Slyngshede:

[operations/puppet@production] P:kerberos::replication absent Icinga check

https://gerrit.wikimedia.org/r/1003383

gerritbot added a comment.Feb 27 2024, 8:15 AM

Change 1003382 merged by Slyngshede:

[operations/puppet@production] P:kerberos::kdc absent Icinga check

https://gerrit.wikimedia.org/r/1003382

SLyngshede-WMF updated the task description. (Show Details)Feb 27 2024, 8:16 AM
gerritbot added a comment.Feb 27 2024, 11:45 AM

Change 1006907 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:pki::multirootca::monitoring Collect metrics from intermediate.

https://gerrit.wikimedia.org/r/1006907

gerritbot added a comment.Feb 27 2024, 1:24 PM

Change 1006907 merged by Slyngshede:

[operations/puppet@production] P:pki::multirootca::monitoring Collect metrics from intermediate.

https://gerrit.wikimedia.org/r/1006907

gerritbot added a comment.Feb 28 2024, 12:22 PM

Change 1007321 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/alerts@master] PKI: Switch alerts to use the x509 metric.

https://gerrit.wikimedia.org/r/1007321

gerritbot added a comment.Feb 28 2024, 12:24 PM

Change 1004061 abandoned by Slyngshede:

[operations/puppet@production] C:ganeti::prometheus Collect process information

Reason:

Abandoned in favor of a more general solution: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1004672

https://gerrit.wikimedia.org/r/1004061

gerritbot added a comment.Feb 29 2024, 6:06 PM

Change 979902 abandoned by Slyngshede:

[operations/puppet@production] C:prometheus::node_exporter allow CPU flags collection

Reason:

Monitoring of CPU flags is no longer required.

https://gerrit.wikimedia.org/r/979902

gerritbot added a comment.Mar 1 2024, 7:47 AM

Change 1005736 merged by jenkins-bot:

[operations/alerts@master] Puppetmaster: Adjust alerting interval.

https://gerrit.wikimedia.org/r/1005736

gerritbot added a comment.Mar 4 2024, 8:40 AM

Change 1004619 merged by jenkins-bot:

[operations/alerts@master] Load average for Swift cluster

https://gerrit.wikimedia.org/r/1004619

gerritbot added a comment.Mar 4 2024, 9:00 AM

Change 1007321 merged by jenkins-bot:

[operations/alerts@master] PKI: Switch alerts to use the x509 metric.

https://gerrit.wikimedia.org/r/1007321

fgiunchedi edited projects, added: SRE Observability (FY2023/2024-Q4); removed: SRE Observability (FY2023/2024-Q3).Mar 26 2024, 2:57 PM
gerritbot added a comment.May 10 2024, 4:09 PM

Change #995180 abandoned by Btullis:

[operations/puppet@production] P:hue absent Icinga monitoring.

Reason:

Hue has been decommissioned.

https://gerrit.wikimedia.org/r/995180

SLyngshede-WMF updated the task description. (Show Details)May 13 2024, 6:42 AM
gerritbot added a comment.May 15 2024, 10:45 AM

Change #1031834 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:ganeti Prometheus monitoring of ganeti services.

https://gerrit.wikimedia.org/r/1031834

gerritbot added a comment.May 21 2024, 7:12 AM

Change #1031834 merged by Slyngshede:

[operations/puppet@production] P:ganeti Prometheus monitoring of ganeti noded services.

https://gerrit.wikimedia.org/r/1031834

fgiunchedi edited projects, added: SRE Observability (FY2024/2025-Q1); removed: SRE Observability (FY2023/2024-Q4).Jul 3 2024, 8:10 AM
fgiunchedi mentioned this in T370526: Remove load_average check for ms-be/thanos-be.Sep 24 2024, 1:09 PM
gerritbot added a comment.Oct 30 2024, 12:31 PM

Change #1084758 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/alerts@master] Start migrating Netbox alerts from Icinga.

https://gerrit.wikimedia.org/r/1084758

• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q1) board.Oct 31 2024, 9:26 PM
• lmata edited projects, added: SRE Observability (FY2024/2025-Q2); removed: SRE Observability (FY2024/2025-Q1).Nov 5 2024, 5:04 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q2) board.
• lmata moved this task from Epics In Progress to Radar on the SRE Observability (FY2024/2025-Q2) board.Nov 5 2024, 5:07 PM
gerritbot added a comment.Nov 13 2024, 9:14 AM

Change #1084758 merged by jenkins-bot:

[operations/alerts@master] Start migrating Netbox alerts from Icinga.

https://gerrit.wikimedia.org/r/1084758

gerritbot added a comment.Nov 13 2024, 3:37 PM

Change #1090875 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/alerts@master] Netbox: Update runbook, add dashboard and physicalhosts report.

https://gerrit.wikimedia.org/r/1090875

gerritbot added a comment.Nov 14 2024, 7:34 AM

Change #1090875 merged by jenkins-bot:

[operations/alerts@master] Netbox: Update runbook, add dashboard and physicalhosts report.

https://gerrit.wikimedia.org/r/1090875

gerritbot added a comment.Nov 19 2024, 10:14 AM

Change #1092779 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/alerts@master] Netbox alerting: Add remaining netbox alert.

https://gerrit.wikimedia.org/r/1092779

• lmata edited projects, added: SRE Observability (FY2024/2025-Q3); removed: SRE Observability (FY2024/2025-Q2).Jan 14 2025, 3:55 PM
• lmata moved this task from Inbox to Radar on the SRE Observability (FY2024/2025-Q3) board.
fgiunchedi updated the task description. (Show Details)Jan 27 2025, 2:04 PM

I couldn't find the replacement for check microcode, it looks like we have the metric in prometheus though we don't alert

SLyngshede-WMF added a comment.Jan 27 2025, 2:07 PM

@fgiunchedi I think that was a special case check for some hardware we no longer run.

fgiunchedi added a comment.Jan 27 2025, 2:28 PM
In T350694#10497028, @SLyngshede-WMF wrote:

@fgiunchedi I think that was a special case check for some hardware we no longer run.

Ok thank you! I'll followup with a patch to absent the check

gerritbot added a comment.Jan 27 2025, 2:31 PM

Change #1114391 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] base: absent check_microcode

https://gerrit.wikimedia.org/r/1114391

gerritbot added a comment.Jan 28 2025, 8:38 AM

Change #1114391 merged by Filippo Giunchedi:

[operations/puppet@production] base: absent check_microcode

https://gerrit.wikimedia.org/r/1114391

fgiunchedi updated the task description. (Show Details)Jan 28 2025, 8:59 AM
fgiunchedi updated the task description. (Show Details)Jan 28 2025, 12:24 PM
fgiunchedi updated the task description. (Show Details)Jan 28 2025, 12:36 PM
gerritbot added a comment.Feb 27 2025, 8:37 AM

Change #994172 merged by Slyngshede:

[operations/puppet@production] P:systemd::timesyncd absent monitoring, handled by AlertManager

https://gerrit.wikimedia.org/r/994172

gerritbot added a comment.Mar 5 2025, 1:06 PM

Change #1124776 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] sre: limit netbox reports alerts to eqiad and codfw

https://gerrit.wikimedia.org/r/1124776

gerritbot added a comment.Mar 5 2025, 1:40 PM

Change #1124776 merged by Filippo Giunchedi:

[operations/alerts@master] sre: limit netbox reports alerts to eqiad and codfw

https://gerrit.wikimedia.org/r/1124776

fgiunchedi closed subtask T353457: Karma UI shows duplicate alerts as Resolved.Mar 6 2025, 2:27 PM
gerritbot added a comment.Mar 10 2025, 2:17 PM

Change #994164 merged by Slyngshede:

[operations/puppet@production] P:firewall absent conntrack_table_size monitoring.

https://gerrit.wikimedia.org/r/994164

gerritbot added a comment.Mar 11 2025, 9:21 AM

Change #1126503 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:firewall absent conntrack_table_size monitoring.

https://gerrit.wikimedia.org/r/1126503

fgiunchedi mentioned this in T388506: Implement a less noisy way to remove nrpe checks (without UNKNOWN spam).Mar 11 2025, 9:41 AM
gerritbot added a comment.Mar 17 2025, 8:53 AM

Change #1126503 merged by Slyngshede:

[operations/puppet@production] P:firewall absent conntrack_table_size monitoring.

https://gerrit.wikimedia.org/r/1126503

gerritbot added a comment.Mar 17 2025, 1:14 PM

Change #1128405 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:firewall remove connection tracking monitoring

https://gerrit.wikimedia.org/r/1128405

gerritbot added a comment.Mar 19 2025, 7:40 AM

Change #1128405 merged by Slyngshede:

[operations/puppet@production] P:firewall remove connection tracking monitoring

https://gerrit.wikimedia.org/r/1128405

gerritbot added a comment.Mar 20 2025, 1:34 PM

Change #1129827 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:mirrors add file age exporter

https://gerrit.wikimedia.org/r/1129827

gerritbot added a comment.Mar 24 2025, 1:22 PM

Change #1129827 merged by Slyngshede:

[operations/puppet@production] P:mirrors add file age exporter

https://gerrit.wikimedia.org/r/1129827

gerritbot added a comment.Mar 24 2025, 1:27 PM

Change #1130594 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:firewall remove unused code

https://gerrit.wikimedia.org/r/1130594

gerritbot added a comment.Mar 24 2025, 1:51 PM

Change #1130594 merged by Slyngshede:

[operations/puppet@production] P:firewall remove unused code

https://gerrit.wikimedia.org/r/1130594

gerritbot added a comment.Mar 25 2025, 9:48 AM

Change #1130964 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/alerts@master] Alert when mirrors become out of date

https://gerrit.wikimedia.org/r/1130964

gerritbot added a comment.Mar 26 2025, 7:53 AM

Change #1130964 merged by jenkins-bot:

[operations/alerts@master] Alert when mirrors become out of date

https://gerrit.wikimedia.org/r/1130964

gerritbot added a comment.Mar 26 2025, 8:05 AM

Change #1131259 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:systemd::timesyncd remove deprecated Icinga check

https://gerrit.wikimedia.org/r/1131259

gerritbot added a comment.Mar 26 2025, 8:51 AM

Change #1131259 merged by Slyngshede:

[operations/puppet@production] P:systemd::timesyncd remove deprecated Icinga check

https://gerrit.wikimedia.org/r/1131259

SLyngshede-WMF updated the task description. (Show Details)Mar 26 2025, 8:57 AM
gerritbot added a comment.Mar 26 2025, 12:24 PM

Change #1092779 merged by jenkins-bot:

[operations/alerts@master] Netbox alerting: Add remaining netbox alert.

https://gerrit.wikimedia.org/r/1092779

fgiunchedi removed a project: SRE Observability (FY2024/2025-Q3).Apr 3 2025, 3:37 PM
SLyngshede-WMF added a comment.Apr 9 2025, 10:12 AM

Regarding the disk space alert: The current AlertManager check will trigger on 6% free disk space, while the Icinga checks are configurable.

In the Puppet repo we have ranges from warning at 15% down to 6% and critical from 10% down to 3%. Some hosts also have different levels for different storage types, on the same host.

So see the Icinga configs in the Puppet repository do:

grep -r nrpe_check_disk_options *
fgiunchedi added a comment.Apr 9 2025, 12:15 PM
In T350694#10725271, @SLyngshede-WMF wrote:

Regarding the disk space alert: The current AlertManager check will trigger on 6% free disk space, while the Icinga checks are configurable.

In the Puppet repo we have ranges from warning at 15% down to 6% and critical from 10% down to 3%. Some hosts also have different levels for different storage types, on the same host.

So see the Icinga configs in the Puppet repository do:

grep -r nrpe_check_disk_options *

Thank you, I have asked an LLM to diff the options with the standard ones and sum up the results (take it with a grain of salt. I did a spot check and the results check out) https://phabricator.wikimedia.org/P74815

gerritbot added a comment.Apr 16 2025, 11:57 AM

Change #1136989 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/alerts@master] Netbox: simplify query

https://gerrit.wikimedia.org/r/1136989

gerritbot added a comment.Apr 23 2025, 8:57 AM

Change #1136989 merged by Slyngshede:

[operations/alerts@master] Netbox: simplify query

https://gerrit.wikimedia.org/r/1136989

gerritbot added a comment.Jun 10 2025, 8:55 AM

Change #1155140 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] monitoring services: add migration task T350694 to instances

https://gerrit.wikimedia.org/r/1155140

gerritbot added a comment.Jun 23 2025, 1:09 PM

Change #1155140 merged by Tiziano Fogli:

[operations/puppet@production] monitoring services: add migration task T350694 to instances

https://gerrit.wikimedia.org/r/1155140

tappof changed the subtype of this task from "Task" to "Goal".Sep 2 2025, 1:33 PM
tappof updated the task description. (Show Details)Sep 2 2025, 1:53 PM
tappof updated the task description. (Show Details)Sep 3 2025, 7:21 AM
gerritbot added a comment.Sep 3 2025, 7:25 AM

Change #1184447 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] base: remove check_microcode

https://gerrit.wikimedia.org/r/1184447

gerritbot added a comment.Oct 13 2025, 2:55 PM

Change #1195712 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] monitoring services: add migration task T350694 to instances

https://gerrit.wikimedia.org/r/1195712

gerritbot added a comment.Oct 13 2025, 3:14 PM

Change #1195712 merged by Tiziano Fogli:

[operations/puppet@production] monitoring services: add migration task T350694 to instances

https://gerrit.wikimedia.org/r/1195712

gerritbot added a comment.Oct 13 2025, 4:02 PM

Change #1195735 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] monitoring services: add migration task T350694 to instances

https://gerrit.wikimedia.org/r/1195735

gerritbot added a comment.Oct 13 2025, 4:09 PM

Change #1195735 merged by Tiziano Fogli:

[operations/puppet@production] monitoring services: add migration task T350694 to instances

https://gerrit.wikimedia.org/r/1195735

tappof updated the task description. (Show Details)Oct 15 2025, 1:13 PM
tappof mentioned this in T384472: Candidate nrpe checks for compatibility layer icinga/prometheus/alertmanager.Oct 17 2025, 3:54 PM
gerritbot added a comment.Oct 30 2025, 2:13 PM

Change #1200074 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] dbctl: enable nrpe2nodexp wrapper on dbctl_uncommitted_diffs

https://gerrit.wikimedia.org/r/1200074

gerritbot added a comment.Oct 30 2025, 3:06 PM

Change #1184447 merged by Tiziano Fogli:

[operations/puppet@production] base: remove check_microcode

https://gerrit.wikimedia.org/r/1184447

gerritbot added a comment.Oct 30 2025, 3:06 PM

Change #1200074 merged by Tiziano Fogli:

[operations/puppet@production] dbctl: enable nrpe2nodexp wrapper on dbctl_uncommitted_diffs

https://gerrit.wikimedia.org/r/1200074

gerritbot added a comment.Oct 31 2025, 2:48 PM

Change #1200365 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] netbox: enable nrpe2nodexp wrapper on check_uncommitted_dns_changes check

https://gerrit.wikimedia.org/r/1200365

gerritbot added a comment.Nov 12 2025, 1:28 PM

Change #1200365 merged by Tiziano Fogli:

[operations/puppet@production] netbox: enable nrpe2nodexp wrapper on check_uncommitted_dns_changes check

https://gerrit.wikimedia.org/r/1200365

gerritbot added a comment.Feb 4 2026, 2:37 PM

Change #1003374 abandoned by Slyngshede:

[operations/puppet@production] P:ganeti: Absent checks for generic Ganeti services.

https://gerrit.wikimedia.org/r/1003374

gerritbot added a comment.Mar 12 2026, 1:24 PM

Change #1251080 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove obsolete Icinga check

https://gerrit.wikimedia.org/r/1251080

gerritbot added a comment.Mar 17 2026, 12:01 PM

Change #1251080 merged by Muehlenhoff:

[operations/puppet@production] Remove obsolete Icinga check

https://gerrit.wikimedia.org/r/1251080

hnowlan mentioned this in T407117: Move remaining TCP/HTTP/HTTPS checks to blackbox.Jul 3 2026, 11:23 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits