Experience in running ChangeProp in production showed, that kafka-node driver doesn't meet stability or performance requirements. Alternative existing driver, kafka-native doesn't support kafka 0.9 and doesn't seem to be actively maintained. So, after a lot of consideration it's been decided to try building our own kafka driver for node.js. A very early prototype could be found here.

Currently it's a wrapper over librdkafka C++ API, that supports asynchronous message consuming and fire-and-forget production of the messages. The effort is still in a very early stage, but performance already seem promising, on my laptop I could reach consumption rate of ~100.000 msg/s.

Obviously, this is still very early to judge about the stability of this driver, and a lot of work still needs to be done. Here's a brief outline of the plan:

• Support providing the configuration
• Support fire-and-forget asynchronous commits

After that's done, it would be possible to reimplement ChangeProp on top of the new driver and test whether it fits our needs. If we find the approach good enough, some of the next steps would be:

• Support callbacks on production
• Support callbacks on offset commits
• Properly handle all the error conditions
• Review locking and thread-safety, optimise for performance, write tests etc.

I still cannot estimate the scope of work needed to be done or write any timeline on this, because it's still not clear how CP would perform on this new driver