Page MenuHomePhabricator

Review the most critical/popular Kafka clients before the Kafka upgrade
Closed, ResolvedPublic

Description

As discussed in the SIG meeting, we should review the most popular/critical Kafka clients for every cluster and check if they are running an up-to-date version of their libraries or not. To better segment the scope of the task and avoid spending hours and hours on it, we should probably pick up max 2/3 clients for each cluster (jumbo, logging, main) and report their status in the task.

Most of the clients should use librdkafka behind the scenes, let's make sure that the versions that we run are not too out-of-date.

We can use the Google doc that Balthazar created as starting point (there is a section dedicated to the clients): https://docs.google.com/document/d/1eHqkgKZitERH3M4NkJPA3qW3qWqF4haIkyMbZLJtATw/edit?tab=t.0

Event Timeline

elukey triaged this task as Medium priority.Feb 12 2026, 9:49 AM

We have checked all PyFlink applications, they are using Flink 1.20 and JDK 17. On a test on the Kafka Test cluster, they worked fine during the migration to Kafka 3.7.

All the eventgate-* applications are using librdkafka 2.2.0 internally and should be fine to work with Kafka 3.7.

Gobblin doesn't specify which library/version is using, but we are using one of the latest Gobblin versions, and it worked fine on the Kafka Test cluster in version 3.7.

This has been handled out of band, and is no longer necessary to keep open now that we're performing the upgrade (or have done so, depending on the cluster).

brouberol claimed this task.
brouberol removed brouberol as the assignee of this task.