Change Details

The [kafka-node](https://github.com/d00rman/kafka-node) library lacks thorough error handling & recovery, and does not support Kafka 0.10. Shortcomings in fault handling have contributed to at least one outage. To address this, we should look into replacing it with a better kafka binding. The [librdkafka](https://github.com/edenhill/librdkafka) C library is well-maintained, and is what we are investigating for Python as well. There are node bindings calle [node-kafka-native](https://github.com/alfred-landrum/node-kafka-native), which look like a good starting point. There is active development, but we'll have to check how mature it really is. ## Requirement: Load balancing between consumers in a single-partition consumer group For ChangeProp, we have a specific requirement for balancing single-partition topics across many consumers in the same consumer group. We have implemented this on top of kafka-node with a randomized take-over algorithm, which talks directly to Zookeeper. We'll have to provide similar functionality in the new driver.

The [kafka-node](https://github.com/d00rman/kafka-node) library lacks thorough error handling & recovery, and does not support Kafka 0.10+. Shortcomings in fault handling have contributed to at least one outage. To address this, we should look into replacing it with a better kafka binding. The [librdkafka](https://github.com/edenhill/librdkafka) C library is well-maintained, and is what we are investigating for Python as well. There are node bindings calle [node-kafka-native](https://github.com/alfred-landrum/node-kafka-native), which look like a good starting point. There is active development, but we'll have to check how mature it really is. ## Requirement: Load balancing between consumers in a single-partition consumer group For ChangeProp, we have a specific requirement for balancing single-partition topics across many consumers in the same consumer group. We have implemented this on top of kafka-node with a randomized take-over algorithm, which talks directly to Zookeeper. We'll have to provide similar functionality in the new driver.

The [kafka-node](https://github.com/d00rman/kafka-node) library lacks thorough error handling & recovery, and does not support Kafka 0.10+. Shortcomings in fault handling have contributed to at least one outage. To address this, we should look into replacing it with a better kafka binding. The [librdkafka](https://github.com/edenhill/librdkafka) C library is well-maintained, and is what we are investigating for Python as well. There are node bindings calle [node-kafka-native](https://github.com/alfred-landrum/node-kafka-native), which look like a good starting point. There is active development, but we'll have to check how mature it really is. ## Requirement: Load balancing between consumers in a single-partition consumer group For ChangeProp, we have a specific requirement for balancing single-partition topics across many consumers in the same consumer group. We have implemented this on top of kafka-node with a randomized take-over algorithm, which talks directly to Zookeeper. We'll have to provide similar functionality in the new driver.