One piece of the puzzle here is that some/most checks are in Icinga while others are in prometheus.

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=Categories+update

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=wdqs

The "Status: Unknown" combined with "NRPE: Unable to read output" usually means that the nagios-nrpe-server process on the affected machines crashed and needs a restart.

This is common when machines run out of RAM and then oom-killer just likes to kill NRPE first.

So I suggest first do a sudo systemctl restart nagios-nrpe-server on those machines with the "unknown" alerts first and see if that resolves them.

If that's the case maybe what is happening is that the data transfer cookbook caused machines to be out of RAM .. or something else.

Separate from that there is also the part about notifications.

For some reason Icinga stopped sending emails (T253733) quite some time ago and unless you have custom notification methods in Icinga that probably means the only place where alerts happen is on IRC in the -operations channel. Which likely means it's not very effective at actually alerting specific people.

Some possible fixes could be:

• create a contact in Icinga config using a team or list email address
• configure Icinga checks to associate this contact with existing checks

• migrate checks from Icinga to Prometheus / alert_manager (possible for http(s) checks but likely not for things like process checks via NRPE)

• for existing and hypothetically migrated checks:
  • use existing or add new receiver together with prometheus::blackbox::http checks
  • configure this receiver to send mail to a list or team address and / or to create Phabricator tickets as notification method

The latter can be copied for example from the sre-collab team setup.

Update: Since the public categories endpoint does not route to the categories instances hosted on wdqs-main and wdqs-scholarly hosts, there is no point in loading categories as no clients will ever reach these instances.

As such, we've elected to remove the Puppet code that installs categories on the graph split hosts, and it's very likely we'll move categories to its own infrastructure (see T374016 ).

Upon further review, it looks like there are still some lingering services related to categories on the graph split hosts:

root@wdqs2021:~# systemctl list-units | grep -i categories
  prometheus-blazegraph-exporter-wdqs-categories.service                                   loaded active running   Prometheus Blazegraph Exporter (wdqs-categories)
  wdqs-categories.service                                                                  loaded active running   Query Service - Blazegraph - wdqs-categories
  load-categories-daily.timer                                                              loaded active waiting   Periodic execution of load-categories-daily.service
  wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-categories.timer                    loaded active waiting   Periodic execution of wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-categories.service

I stopped the services and ran puppet:

root@wdqs2021:~# systemctl disable --now prometheus-blazegraph-exporter-wdqs-categories.service wdqs-categories.service load-categories-daily.timer wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-categories.timer
root@wdqs2021:~# run-puppet-agent

Unfortunately, this left the wdqs-categories service in a failed state. Re-opening so we can figure out the best way to remove the categories service and its exporters.

Mentioned in SAL (#wikimedia-operations) [2024-09-13T16:56:14Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on wdqs[1021-1024].eqiad.wmnet with reason: T373935

Mentioned in SAL (#wikimedia-operations) [2024-09-13T16:56:31Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wdqs[1021-1024].eqiad.wmnet with reason: T373935