Kafka jumbo is currently running 1.0.0. [[ https://docs.confluent.io/current/release-notes.html#apache-kafka-1-0-1-cp1 | 1.0.1 contains some bugfixes ]], and 1.1.0 has a couple of nice looking changes too ([[ https://docs.confluent.io/current/release-notes.html#reduced-producer-latency | reduced latency for produce requests ]], [[ https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration | dynamic broker configs ]], and [[ https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74686649 | improved consumer lag jmx metrics ]].
I'd like to upgrade main Kafka's to 1.1.0, but before doing so, we should probably upgrade jumbo first.
https://kafka.apache.org/documentation/#upgrade_1_1_0
# Upgrade plan
Go to https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=Kafka+Broker+Under+Replicated+Partitions and schedule downtime for other brokers in the cluster you are working on.
1. Add confluent-kafka-2.11=1.1.0 to apt.wikimedia.org
2. Install confluent-kafka-2.11=1.1.0, remove unwanted systemd units and directories:
```
sudo apt-get install confluent-kafka-2.11
sudo rm -rv /var/log/confluent /var/lib/kafka /var/lib/zookeeper /lib/systemd/system/confluent*.service
```
3. Restart each broker one at a time. (`inter.broker.protocol.version` is already set to 1.0.0.)
```
sudo service kafka restart
# wait until webrequest_text gets leaders for this broker again:
watch -n 5 'id=$(sudo grep broker.id= /etc/kafka/server.properties | awk -F "=" "{print \$2}"); echo "Leader: $id?"; kafka topics --describe --topic webrequest_text | grep "Leader: $id"'
```
4. Set `inter.broker.protocol.version=1.1`: https://gerrit.wikimedia.org/r/#/c/430398/
5. restart each broker one at a time.