The kafkamon hosts need to be upgraded to Buster. This is the actual status:
elukey@kafkamon1001:~$ dpkg -l | grep burrow ii burrow 1.1.0-1 amd64 Kafka Burrow ii prometheus-burrow-exporter 0.0.5-1 amd64 Prometheus exporter for the Kafka Burrow daemon.
It is very nice that Burrow is packaged for Buster: https://packages.debian.org/buster/burrow, and it is version 1.2.1. The prometheus exporter doesn't seem to be in Debian yet, but we are not far from the last upstream (https://github.com/jirwin/burrow_exporter/releases) so I'd say we'd just need to rebuild for Stretch (or copy the package, IIRC we built it for sid creating a single go binary blob).
What I would do then is the following:
- Create kafkamon1002 and kafkamon2002 with Buster.
- Temporary fork role::kafka::monitoring to role::kafka::monitoring_buster (or a similar approach) to avoid scraping duplicate metrics from the new hosts while they are being prepared.
- Deploy Burrow via Debian upstream package, configure the host and see if it works well.
- Once we are ready, we switch the prometheus masters to the new hosts.
- Clean up forked role
- We drop kafkamon1001/2001.
Since this is some work that is shared between Analytics and SRE, should we try to do this work together to speed it up? Thoughts?