Page MenuHomePhabricator
Create Task
Maniphest T424527

Upgrade kafka-jumbo to kafka 3.7
Closed, ResolvedPublic
Actions

Description

  • Pin the inter broker protocol version on the brokers to hieradata/role/common/kafka/jumbo/broker.yaml:profile::kafka::broker::inter_broker_protocol_version: 1.1.0
  • Perform a rolling upgrade of the brokers, that will restart with the pinned version configurations and the new kafka version, using host-by-host patches and service restart of kafka broker, e.g. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1273863
    • kafka-jumbo1010
    • kafka-jumbo1011
    • kafka-jumbo1012
    • kafka-jumbo1013
    • kafka-jumbo1014
    • kafka-jumbo1015
    • kafka-jumbo1017
    • kafka-jumbo1018
    • kafka-jumbo1016 (controller)
  • Change the inter broker protocol version to match the new kafka version

    Set hieradata/role/common/kafka/logging.yaml:profile::kafka::broker::inter_broker_protocol_version: 3.7
  • Perform a final rolling restart of the brokers

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
kafka-jumbo: set inter.broker.protocol to 3.7operations/puppetproduction+1 -1
kafka-jumbo: deploy kafka 3.7 to all brokersoperations/puppetproduction+1 -2
kafka-jumbo: update kafka-jumbo1010 confluent distro to 77operations/puppetproduction+1 -0
kafka-jumbo: update kafka-jumbo1010 confluent distro to 77operations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT416669 Upgrade Kafka to version 3.x
 ResolvedbrouberolT424527 Upgrade kafka-jumbo to kafka 3.7

Event Timeline

brouberol created this task.Apr 27 2026, 2:07 PM
brouberol triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 27 2026, 2:07 PM
gerritbot added a comment.Apr 27 2026, 2:08 PM

Change #1277558 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] kafka-jumbo: update kafka-jumbo1010 confluent distro to 77

https://gerrit.wikimedia.org/r/1277558

gerritbot added a project: Patch-For-Review.Apr 27 2026, 2:09 PM
brouberol updated the task description. (Show Details)Apr 27 2026, 2:09 PM
gerritbot added a comment.Apr 27 2026, 2:10 PM

Change #1277555 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] kafka-jumbo: update kafka-jumbo1010 confluent distro to 77

https://gerrit.wikimedia.org/r/1277555

gerritbot added a comment.Apr 27 2026, 2:10 PM

Change #1277558 abandoned by Brouberol:

[operations/puppet@production] kafka-jumbo: update kafka-jumbo1010 confluent distro to 77

Reason:

Created by mistake

https://gerrit.wikimedia.org/r/1277558

gerritbot added a comment.Apr 27 2026, 2:11 PM

Change #1277555 merged by Brouberol:

[operations/puppet@production] kafka-jumbo: update kafka-jumbo1010 confluent distro to 77

https://gerrit.wikimedia.org/r/1277555

brouberol changed the task status from Open to In Progress.Apr 27 2026, 2:15 PM
brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
brouberol edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE.
brouberol added a comment.Apr 27 2026, 2:21 PM

kafka-jumbo1010 is now running kafka 3.7, without hurdles!

brouberol@kafka-jumbo1010:~$ dpkg -l | grep confluent-kafka | grep ii
ii  confluent-kafka                      7.7.7-1                            all          publish-subscribe messaging rethought as a distributed commit log

image.png (2,922×2,168 px, 404 KB)

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2026, 2:31 PM
brouberol updated the task description. (Show Details)Apr 27 2026, 2:49 PM
gerritbot added a comment.Apr 28 2026, 8:33 AM

Change #1278355 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] kafka-jumbo: deploy kafka 3.7 to all brokers

https://gerrit.wikimedia.org/r/1278355

gerritbot added a project: Patch-For-Review.Apr 28 2026, 8:33 AM
gerritbot added a comment.Apr 28 2026, 12:51 PM

Change #1278355 merged by Brouberol:

[operations/puppet@production] kafka-jumbo: deploy kafka 3.7 to all brokers

https://gerrit.wikimedia.org/r/1278355

brouberol updated the task description. (Show Details)Apr 28 2026, 1:00 PM
brouberol updated the task description. (Show Details)
brouberol updated the task description. (Show Details)Apr 28 2026, 1:25 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 28 2026, 1:31 PM
brouberol claimed this task.Apr 28 2026, 2:10 PM
brouberol updated the task description. (Show Details)Apr 28 2026, 2:32 PM
brouberol updated the task description. (Show Details)Apr 28 2026, 2:47 PM
brouberol updated the task description. (Show Details)Apr 28 2026, 3:13 PM
brouberol updated the task description. (Show Details)Apr 28 2026, 8:07 PM

Leaving the cluster in a semi-upgraded state caused an incident: https://docs.google.com/document/d/17BuSP9-tlfHN0MRTifP5EFLk1_gS5NaHKi0eo4Jw7JM/edit?tab=t.0#heading=h.jev75r1ysugd

We had to speed up the upgrade process, so all brokers are running 3.7 now!

gerritbot added a comment.Apr 30 2026, 7:18 AM

Change #1280078 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] kafka-jumbo: set inter.broker.protocol to 3.7.0

https://gerrit.wikimedia.org/r/1280078

gerritbot added a project: Patch-For-Review.Apr 30 2026, 7:18 AM
brouberol moved this task from In Progress to To Be Deployed on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.Apr 30 2026, 7:58 AM
gerritbot added a comment.Apr 30 2026, 9:03 AM

Change #1280078 merged by Brouberol:

[operations/puppet@production] kafka-jumbo: set inter.broker.protocol to 3.7

https://gerrit.wikimedia.org/r/1280078

Maintenance_bot removed a project: Patch-For-Review.Apr 30 2026, 9:30 AM
brouberol added a comment.Apr 30 2026, 9:47 AM

We've had to resort to rolling restart the cluster manually, due to a permission error preventing the cookbook from triggering an election (probably due to the jumbo ACLs).

----- OUTPUT for command #1: 'source /etc/prof...topic-partitions' -----
kafka-leader-election --bootstrap-server kafka-jumbo1010.eqiad.wmnet:9092,kafka-jumbo1011.eqiad.wmnet:9092,kafka-jumbo1012.eqiad.wmnet:9092,kafka-jumbo1013.eqiad.wmnet:9092,kafka-jumbo1014.eqiad.wmnet:9092,kafka-jumbo1015.eqiad.wmnet:9092,kafka-jumbo1016.eqiad.wmnet:9092,kafka-jumbo1017.eqiad.wmnet:9092,kafka-jumbo1018.eqiad.wmnet:9092 --election-type PREFERRED --all-topic-partitions
Not authorized to perform leader election
Not authorized to perform leader election
org.apache.kafka.server.common.AdminCommandFailedException: Not authorized to perform leader election
        at org.apache.kafka.tools.LeaderElectionCommand.electLeaders(LeaderElectionCommand.java:134)
        at org.apache.kafka.tools.LeaderElectionCommand.run(LeaderElectionCommand.java:117)
        at org.apache.kafka.tools.LeaderElectionCommand.mainNoExit(LeaderElectionCommand.java:71)
        at org.apache.kafka.tools.LeaderElectionCommand.main(LeaderElectionCommand.java:66)

That being said, the cluster now has inter.broker.protocol: 3.7 enabled everywhere.

I'm calling this done!

brouberol closed this task as Resolved.Apr 30 2026, 9:47 AM
brouberol moved this task from To Be Deployed to Done on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
brouberol updated the task description. (Show Details)Apr 30 2026, 9:48 AM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.May 1 2026, 7:20 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits