Starting point: https://docs.google.com/document/d/1eHqkgKZitERH3M4NkJPA3qW3qWqF4haIkyMbZLJtATw/edit?tab=t.0
We should create a cookbook to upgrade single kafka brokers and/or an entire cluster to Kafka 3.5 safely. Ideally this work will be reused in the future to ease migrations to future versions, without waiting years to do it like we have been done for Kafka 1.1 :D
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T416669 Upgrade Kafka to version 3.x | |||
| Resolved | elukey | T417035 Create a cookbook to execute Kafka rolling upgrades |
From the tests in T416670, the procedure should be:
Note: rollout and rollback are the same, the only thing that changes is the puppet patch. It is questionable if we want to have a full rollback solution or just allow the user to file a new puppet patch and restart the cookbook.
At this point the cluster is running the new kafka code, but inter.broker.protocol.version is still using the old value. Due to a change happened in the way the schema related to the clients' offset schema, rolling back is not possible after changing this value. So the user should likely wait a couple of days before proceeding further.
Then, it is sufficient to merge the inter.broker.protocol.version upgrade and roll restart brokers one at the time.
So ideally two cookbooks, very simple ones.
Change #1247942 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/cookbooks@master] WIP: add sre.kafka.change-confluent-distro-version
Change #1248430 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] role::kafka::test::broker: upgrade Kafka to 3.5
Change #1248430 merged by Elukey:
[operations/puppet@production] role::kafka::test::broker: upgrade Kafka to 3.5
Change #1247942 merged by Elukey:
[operations/cookbooks@master] Add the sre.kafka.change-confluent-distro-version cookbook
Change #1249940 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] role::kafka::test::broker: move to Confluent Kafka 3.7
Change #1249940 merged by Elukey:
[operations/puppet@production] role::kafka::test::broker: move to Confluent Kafka 3.7
Change #1262008 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] role::kafka::test: update the inter broker protocol
Change #1262008 merged by Elukey:
[operations/puppet@production] role::kafka::test: update the inter broker protocol
Change #1262031 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] Move kafka-test1006 to Trixie
Change #1262031 merged by Elukey:
[operations/puppet@production] Move kafka-test1006 to Trixie