As part of the Jumbo cluster upgrade, we've started using the --new.consumer option of the 0.9 MirrorMaker process. The new consumer was experimental when 0.9 was released, so it was not the default. Newer MirrorMaker versions have completely removed the old consumer client.
As part of the goal to upgrade Kafka main clusters to 1.x in Q4, we will need to switch to a new consumer client based MirrorMaker at some point. New consumer has been *much* more stable for us when replicating from main-eqiad -> jumbo, especially around consumer rebalances, etc. I've also added a new and better prometheus based dashboard and alerting for the new MirrorMaker instance.
However, the --new.consumer uses Kafka to store offsets instead of zookeeper, so I'm pretty sure this will reset all offsets for MirrorMaker. We need to figure out how best to deal with this for change-prop and job queue. I believe that change-prop does deduplication, so if possible the best thing to do would be to spawn up new consumer MirrorMaker instances in a new consumer group, with a very short overlap before shutting down the old consumer based ones. While both MirrorMaker instances are running, we'd get duplicate messages, but hopefully the # would be limited.