Page MenuHomePhabricator
Create Task
Maniphest T423723

Upgrade kafka-logging to version 3.7
Closed, ResolvedPublic
Actions

Description

High level steps (summarized from docs in T416669: Upgrade Kafka to version 3.x)

Kafka logging has 2 clusters, one for each site of eqiad and codfw.

  • Kafka-logging eqiad:
    • Pin the inter broker protocol version on the brokers to hieradata/role/common/kafka/logging.yaml:profile::kafka::broker::inter_broker_protocol_version: 1.1.0
    • Perform a rolling upgrade of the brokers, that will restart with the pinned version configurations and the new kafka version, using host-by-host patches and service restart of kafka broker, e.g. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1273863
      • kafka-logging1001
      • kafka-logging1002
      • kafka-logging1003
      • kafka-logging1004
      • kafka-logging1005
    • Change the inter broker protocol version to match the new kafka version
      • Set hieradata/role/common/kafka/logging.yaml:profile::kafka::broker::inter_broker_protocol_version: 3.7
    • Perform a final rolling restart of the brokers
  • Kafka-logging codfw:
    • Pin the inter broker protocol version on the brokers to hieradata/role/common/kafka/logging.yaml:profile::kafka::broker::inter_broker_protocol_version: 1.1.0
    • Perform a rolling upgrade of the brokers, that will restart with the pinned version configurations and the new kafka version, using host-by-host patches and service restart of kafka broker, e.g. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1273863
      • kafka-logging2001
      • kafka-logging2002
      • kafka-logging2003
      • kafka-logging2004
      • kafka-logging2005
    • Change the inter broker protocol version to match the new kafka version
      • Set hieradata/role/common/kafka/logging.yaml:profile::kafka::broker::inter_broker_protocol_version: 3.7
    • Perform a final rolling restart of the brokers

Post-upgrade cleanup:

  • Clean up per-host hiera configs, move to role level

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
kafka-logging: set eqiad (and all) brokers to protocol 3.7operations/puppetproduction+1 -1
kafka-logging: set eqiad (and all) brokers to confluent distro 77operations/puppetproduction+2 -1
kafka-logging: set codfw brokers inter-broker protocol to 3.7operations/puppetproduction+1 -0
kafka-logging: set all codfw brokers to confluent_distribution 77operations/puppetproduction+0 -0
kafka-logging: update kafka-logging2001 confluent distro to 77operations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• herron created this task.Apr 17 2026, 4:36 PM
• herron triaged this task as High priority.
• herron updated the task description. (Show Details)
• herron added a parent task: T417001: Upgrade Observability Kafka-logging hosts to trixie.
• herron mentioned this in T417001: Upgrade Observability Kafka-logging hosts to trixie.Apr 17 2026, 4:41 PM

@brouberol, do the steps in the description look ok to you, anything I missed?

• herron claimed this task.Apr 17 2026, 4:42 PM
gerritbot added a comment.Apr 17 2026, 4:49 PM

Change #1273863 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: update kafka-logging2001 confluent distro to 77

https://gerrit.wikimedia.org/r/1273863

gerritbot added a project: Patch-For-Review.Apr 17 2026, 4:49 PM
brouberol added a comment.Apr 17 2026, 5:34 PM

cc @elukey as he was the one working on the puppet side of things. It does look good to me though!

Daimona unsubscribed.Apr 17 2026, 5:55 PM
elukey added a comment.Apr 20 2026, 7:11 AM

@herron I would change a thing - I think it is sufficient to upgrade a single host (like https://gerrit.wikimedia.org/r/c/operations/puppet/+/1273863), leave it there for a couple of days to catch any issues and then upgrade the rest in one go with the cookbook (it doesn't support single hosts or subsets at the moment). It should be very safe, not real difference with the host-by-host approach that may be more tedious and error prone. But I'll leave the choice to you, the plan looks ok as well!

Let us know when you do it so we'll be able to assist/help if needed.

Clement_Goubert moved this task from Inbox to Radar (Pending) on the ServiceOps new board.Apr 20 2026, 8:11 AM
• herron added a subscriber: colewhite.Apr 20 2026, 4:35 PM
In T423723#11837649, @elukey wrote:

@herron I would change a thing - I think it is sufficient to upgrade a single host (like https://gerrit.wikimedia.org/r/c/operations/puppet/+/1273863), leave it there for a couple of days to catch any issues and then upgrade the rest in one go with the cookbook (it doesn't support single hosts or subsets at the moment). It should be very safe, not real difference with the host-by-host approach that may be more tedious and error prone. But I'll leave the choice to you, the plan looks ok as well!

Let us know when you do it so we'll be able to assist/help if needed.

Sounds good! I'll go ahead with kafka-logging2001 today as the first single host and let that settle in for a few days.

gerritbot added a comment.Apr 20 2026, 4:36 PM

Change #1273863 merged by Herron:

[operations/puppet@production] kafka-logging: update kafka-logging2001 confluent distro to 77

https://gerrit.wikimedia.org/r/1273863

• herron updated the task description. (Show Details)Apr 20 2026, 4:42 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 20 2026, 5:32 PM
gerritbot added a comment.Apr 21 2026, 2:05 PM

Change #1275932 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: set all codfw brokers to confluent_distribution 77

https://gerrit.wikimedia.org/r/1275932

gerritbot added a project: Patch-For-Review.Apr 21 2026, 2:05 PM
gerritbot added a comment.Apr 23 2026, 4:11 PM

Change #1275932 merged by Herron:

[operations/puppet@production] kafka-logging: set all codfw brokers to confluent_distribution 77

https://gerrit.wikimedia.org/r/1275932

• herron updated the task description. (Show Details)Apr 23 2026, 4:30 PM

Cookbook worked well!

END (PASS) - Cookbook sre.kafka.change-confluent-distro-version (exit_code=0) Change Confluent distribution for Kafka A:kafka-logging-codfw cluster: Change Confluent distribution.
Maintenance_bot removed a project: Patch-For-Review.Apr 23 2026, 4:31 PM
gerritbot added a comment.Apr 23 2026, 4:38 PM

Change #1276745 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: set codfw brokers inter-broker protocol to 3.7

https://gerrit.wikimedia.org/r/1276745

gerritbot added a project: Patch-For-Review.Apr 23 2026, 4:38 PM
gerritbot added a comment.Apr 27 2026, 2:44 PM

Change #1276745 merged by Herron:

[operations/puppet@production] kafka-logging: set codfw brokers inter-broker protocol to 3.7

https://gerrit.wikimedia.org/r/1276745

• herron renamed this task from Upgrade kafka-logging to version 3.x to Upgrade kafka-logging to version 3.7.Apr 27 2026, 3:17 PM
• herron updated the task description. (Show Details)
• herron updated the task description. (Show Details)
gerritbot added a comment.Apr 27 2026, 3:22 PM

Change #1277581 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: set eqiad (and all) brokers to confluent distro 77

https://gerrit.wikimedia.org/r/1277581

gerritbot added a comment.Apr 28 2026, 2:27 PM

Change #1277581 merged by Herron:

[operations/puppet@production] kafka-logging: set eqiad (and all) brokers to confluent distro 77

https://gerrit.wikimedia.org/r/1277581

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2026, 2:31 PM
gerritbot added a comment.Apr 28 2026, 2:49 PM

Change #1278489 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: set eqiad (and all) brokers to protocol 3.7

https://gerrit.wikimedia.org/r/1278489

gerritbot added a project: Patch-For-Review.Apr 28 2026, 2:49 PM
• herron updated the task description. (Show Details)Apr 28 2026, 2:59 PM
gerritbot added a comment.Apr 28 2026, 3:10 PM

Change #1278489 merged by Herron:

[operations/puppet@production] kafka-logging: set eqiad (and all) brokers to protocol 3.7

https://gerrit.wikimedia.org/r/1278489

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2026, 3:32 PM
• herron updated the task description. (Show Details)Apr 28 2026, 3:34 PM
• herron closed this task as Resolved.Apr 28 2026, 4:47 PM

All kafka-logging brokers have been upgraded to 3.7

hnowlan awarded a token.Apr 29 2026, 11:49 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits