We use Camus to batch import messages from Kafka into HDFS. Camus uses the 0.8.0 old SimpleConsumer to consume from Kafka. However "SSL is supported only for the new Kafka Producer and Consumer, the older API is not supported".
So HM! Options:
- Update Camus to use newer Consumer API with SSL support.
- Import into HDFS with something other than Camus.
We've done some small spikes for 2 before, examining both Kafka Connect and Gobblin. Neither did quite what we need at the time, but perhaps we should take a minute to reexamine these options, before attempting to patch up Camus.