T167039 is about upgrading Kafka main clusters to 1.x. This ticket is about enabling SSL and inter broker encryption after the 1.x upgrade is done.
https://docs.confluent.io/current/kafka/incremental-security-upgrade.html
# Prep work
[x] Kafka upgraded to 1.x T167039
[x] Addition of main Kafka broker TLS keys and certs and ssl_password in hiera.
# production upgrade plan [WIP]
This upgrade requires 2 rolling restarts of each broker in a Kafka cluster.
1. To enable SSL port communication
2. To set `security.inter.broker.protocol=SSL`
1. Merge //GERRIT_URL_TDB//. For each broker, run puppet to enable SSL listener and restart each broker:
```
sudo puppet agent -t
sudo service kafka restart
# wait until broker is back up and in ISRs, initiate election:
watch "kafka topics --describe --topic eqiad.mediawiki.revision-create | grep -E 'Isr:.*1001.*$'"
kafka preferred-replica-election
# Now proceed with next broker...
```
2. Merge //GERRIT_URL_TDB//. For each broker, run puppet to set default `inter.broker.protocol.version=SSL` and restart each broker:
```
sudo puppet agent -t
sudo service kafka restart
# wait until broker is back up and in ISRs, initiate election:
watch "kafka topics --describe --topic eqiad.mediawiki.revision-create | grep -E 'Isr:.*1001.*$'"
kafka preferred-replica-election
# Now proceed with next broker...
```
Done!