As we migrate topics away from brokers kafka-jumbo100[1-6], we encounter the same issue as in https://phabricator.wikimedia.org/T341558: many partitions are empty, causing leadership optimization to cause traffic hotspots. By relying on storage rebalancing + leadership optimization, we should reach a more optimal equilibrium between brokers.
For this, we need to leverage https://github.com/tarvip/kafkakit-prometheus-metricsfetcher, as the stock metricsfetcher relies on the Datadog API. And for this, we need to make it available as a Debian package.