In T421996#11903668, @ecarg wrote:

I believe we have used prometheus-es-exporter to alert on log levels before, here is an example. We then disabled this because they got too noisy and now only alert on rules defined here via Grafana.

I don't think so

I've spent some time experimenting with https://github.com/mahendrapaipuri/grafana-dashboard-reporter-app which looks promising for our use case. This is a Grafana plugin that creates an api endpoint that rendres a dashboard as a pdf report based upon the query string presented to the api. In my view this can do the heavy lifting of generating a visual report, and allows us to do so without straying very far from our existing tooling. We can report on existing dashboards, or create new streamlined report dashboards to be captured on the reporting interval

All kafka-logging hosts have been upgraded to trixie

Thanks for looking @cmooney @elukey! I think we can certainly rule out a network issue. Having a look this morning with fresh eyes, 2005 does appear to be working fine, but very lightly loaded. As in no load at all. I expected 2005 to resync and pull data after reimage, but the host is only responsible for topics/partitions that are presently 0B in size and so, although working, the /srv filesystem is suspiciously empty (2.1M used).

Today I reimaged kafka-logging2005 with --move-vlan and afterwards the node is having trouble rejoining the cluster. I'm seeing errors like this like this logged by other cluster nodes when it attempts to bring the replica up to date:

All kafka-logging brokers have been upgraded to 3.7

Cookbook worked well!

In T423723#11837649, @elukey wrote:

@herron I would change a thing - I think it is sufficient to upgrade a single host (like https://gerrit.wikimedia.org/r/c/operations/puppet/+/1273863), leave it there for a couple of days to catch any issues and then upgrade the rest in one go with the cookbook (it doesn't support single hosts or subsets at the moment). It should be very safe, not real difference with the host-by-host approach that may be more tedious and error prone. But I'll leave the choice to you, the plan looks ok as well!

Let us know when you do it so we'll be able to assist/help if needed.

@brouberol, do the steps in the description look ok to you, anything I missed?

setting low priority as we've just upgrade kafkamon hosts to trixie, so we'll have a Debian upgrade cycle worth of time to perform the move

Low priority question, but since things may have changed since the setup of the current approach I'll ask -- Is smart-data-dump still the better approach for this type of monitoring compared to something along the lines of https://github.com/prometheus-community/smartctl_exporter?

In T196336#11723288, @gerritbot wrote:

Change #1253576 merged by Herron:

[operations/puppet@production] icinga: add monthly restart

https://gerrit.wikimedia.org/r/1253576

At a high level the alerting system is coupled to prometheus metrics. For the purposes of generating metrics from log events we support what's called the promethues-es-exporter which executes queries against logs (opensearch) and transforms the results into metrics which can then be used to generate alerts.

That would be nice! Off hand there's a name convention on the Kafka side as well where clusters are named like logging-eqiad, main-eqiad. I'm not sure if it'd be a problem to change the assumed prefix for the hostname while keeping the same cluster name inside Kafka. Good question for the Kafka SIG!

It could be a different thing. AFAIK sre.hosts.reboot-single sets downtime itself as well, maybe an edge case that happens when downtimes are doubled?

The sloth onboarding backlog is empty!

We chatted about this a bit at the o11y team meeting this week and consensus was that we're looking ok capacity wise, but would like to explore the potential of an SSD backed bucket. I'll give @tappof an opportunity to chime in as well. In the mean time @MatthewVernon what's your initial impression of the SSD idea?

In T419919#11713106, @Volans wrote:

• Each cookbook owners can decide to log the START/END or alternatively a single DONE line using Spicerack API [1] depending if it's a long-lasting or short-lived cookbook.
• Removing those logs from IRC would require a significant workflow change for everyone touching production that right now gets live updates of those on IRC. And AFAIK both wikitech SAL and toolforge SAL don't have any live update mechanism right now. So bypassing IRC to store them directly on SAL is not a solution with the current tooling.