Page MenuHomePhabricator

Upgrade Kafka on jumbo cluster to 1.1.0 (latest)
Closed, ResolvedPublic8 Story Points

Description

Kafka jumbo is currently running 1.0.0. 1.0.1 contains some bugfixes, and 1.1.0 has a couple of nice looking changes too (reduced latency for produce requests, dynamic broker configs, and improved consumer lag jmx metrics.

I'd like to upgrade main Kafka's to 1.1.0, but before doing so, we should probably upgrade jumbo first.

https://kafka.apache.org/documentation/#upgrade_1_1_0

Upgrade plan

Go to https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=Kafka+Broker+Under+Replicated+Partitions and schedule downtime for other brokers in the cluster you are working on.

  1. Add confluent-kafka-2.11=1.1.0 to apt.wikimedia.org
  2. Install confluent-kafka-2.11=1.1.0, remove unwanted systemd units and directories:
sudo apt-get install confluent-kafka-2.11
sudo rm -rv /var/log/confluent /var/lib/kafka /var/lib/zookeeper /lib/systemd/system/confluent*.service
  1. Restart each broker one at a time. (inter.broker.protocol.version is already set to 1.0.0.)
sudo service kafka restart
# wait until webrequest_text gets leaders for this broker again:
watch -n 5 'id=$(sudo grep broker.id= /etc/kafka/server.properties | awk -F "=" "{print \$2}"); echo "Leader: $id?"; kafka topics --describe  --topic webrequest_text | grep "Leader: $id"'
  1. Set inter.broker.protocol.version=1.1: https://gerrit.wikimedia.org/r/#/c/430398/
  2. restart each broker one at a time.

Event Timeline

Ottomata triaged this task as Normal priority.May 1 2018, 2:31 PM
Ottomata created this task.
Ottomata updated the task description. (Show Details)May 1 2018, 2:51 PM

Done in deployment-prep, looks good!

@elukey, everything is working in deployment-prep. Any objections if I start this tomorrow (wednesday?)

elukey added a comment.May 1 2018, 6:17 PM

@elukey, everything is working in deployment-prep. Any objections if I start this tomorrow (wednesday?)

None, let me know if you need help :)

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.May 1 2018, 7:40 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-02T13:49:32Z] <ottomata> beginning upgrade of kafka-jumbo brokers from 1.0.0 -> 1.1.0 : T193495

Mentioned in SAL (#wikimedia-analytics) [2018-05-02T13:49:39Z] <ottomata> beginning upgrade of kafka-jumbo brokers from 1.0.0 -> 1.1.0 : T193495

Ottomata updated the task description. (Show Details)May 2 2018, 2:00 PM
Ottomata updated the task description. (Show Details)May 2 2018, 2:12 PM
Ottomata updated the task description. (Show Details)May 2 2018, 2:27 PM

Change 430398 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] kafka jumbo 1.1.0 inter.broker.protocol.version

https://gerrit.wikimedia.org/r/430398

Change 430398 merged by Ottomata:
[operations/puppet@production] kafka jumbo 1.1.0 inter.broker.protocol.version

https://gerrit.wikimedia.org/r/430398

Ottomata updated the task description. (Show Details)May 2 2018, 3:30 PM
Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.May 2 2018, 4:03 PM
Nuria closed this task as Resolved.May 8 2018, 10:51 PM
Vvjjkkii renamed this task from Upgrade Kafka on jumbo cluster to 1.1.0 (latest) to gwdaaaaaaa.Jul 1 2018, 1:13 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed Ottomata as the assignee of this task.
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed the point value for this task.
Vvjjkkii removed subscribers: Aklapper, gerritbot.
CommunityTechBot set the point value for this task to 8.Jul 1 2018, 3:25 PM
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot renamed this task from gwdaaaaaaa to Upgrade Kafka on jumbo cluster to 1.1.0 (latest).Jul 2 2018, 4:43 PM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to Ottomata.
CommunityTechBot lowered the priority of this task from High to Normal.
CommunityTechBot added subscribers: Aklapper, gerritbot.