For now we're securing the caches' kafka broker traffic with IPSec ( T92602 ), but longer-term we need to move this to TLS.
We'd like to encrypt all Kafka traffic, both inter broker and between clients and brokers.
rough idea
- Make cassandra-ca-manager script generic, and also make it output other useful certificate formats (.pem?) needed by non Java clients.
- Use ca-manager script to generate CA certs, broker keys and client keys needed to configure Kafka TLS.
- Client keys will be generated per logical client, not per client instance. E.g. varnishkafka-webrequest instances will all share the same client keys, distributed by puppet.
- The CA certs will be distributed to all hosts that might run Kafka clients (e.g. Hadoop nodes), to support unauthenticated use cases (still encrypted).
- Logical client keys will be distributed via puppet only to hosts that will run those specific clients.
Note that this plan doesn't yet consider encryption of traffic between Kafka and Zookeeper. Should we?