Until we solve this problem we won't be able to disable PLAINTEXT Kafka use. That means we won't solve the current problem that anyone in prod can consume webrequest data from Kafka.

In the meantime, we could restrict the use of port 9092 (PLAINTEXT, no authentication) to only certain nodes (Hadoop, maybe stat100[24], etc.)

This would allow us to defer working on this task until we figure out a better replacement for Camus, without defeating the purpose of enforcing authentication.

Tasks:
Evaluate whether camus can use newer consumer, if so, proceed this way.

Possible issues:

• Does newer consumer APi manage offsets for us?

How to test:

• We have changes in camus package. We would need to update camus code to run with new kafka library. We can test consuming from kafka and dumping into a different place on HDFS. This can be done before turning in TLS support as that is native to that consumer API mode.