This task is about rebalancing the kafka partitions in main eqiad and codfw after the additional 2 brokers per cluster added some time ago. Kafka doesn't do it by itself, and for Kafka Jumbo we used a special tool that DataDog created called topicmappr. The idea is to:
- use topicmappr to generate a plan (namely json files) representing the moves (like topic X partition 1 from broker 1 to broker 2, etc..).
- use the kafka rebalance partition tool (that gets the above json files as input) to do the actual moves.
The final goal should be to distribute some of the load (traffic etc..) from 3 brokers only to 5 (the newer ones are not doing much at the moment).