Page MenuHomePhabricator
Create Task
Maniphest T399069

Proposal: adding a kafka admin client to spicerack
Closed, ResolvedPublic
Actions

Description

We currently expose a KafkaClient (via Spicerack.kafka, itself a spicerack.kafka.Kafka instance). That client class wraps a kafka consumer, and allows to interact with a topic partitions and associated offset for a given consumer.

I suggest we also allow the Kafka class (scoped to a single cluster) to interact with a KafkaAdminClient, which would allow cookbooks to enquire about the state of the cluster:

  • list active brokers
  • list current partitions
  • list topics
  • etc

This wouldn't require any additional dependencies, as we'd only need to leverage the KafkaAdminClient class provided by kafka-python.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Kafka: expose a KafkaAdminClient builder methodBrouberoloperations/software/spicerackmaster+35 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

brouberol created this task.Jul 9 2025, 10:52 AM
brouberol triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 9 2025, 10:52 AM
brouberol added a project: Spicerack.Jul 9 2025, 10:53 AM
brouberol added a subscriber: Volans.
Restricted Application added a project: SRE-tools. · View Herald TranscriptJul 9 2025, 10:53 AM
brouberol added a comment.Jul 9 2025, 11:06 AM

To illustrate the proposal, this is one of many things you can do with an admin client:

>>> from kafka.admin import KafkaAdminClient
>>> kadmin = KafkaAdminClient(bootstrap_servers='kafka-jumbo1010.eqiad.wmnet:9092')
>>> kadmin.describe_cluster()
{'throttle_time_ms': 0, 'brokers': [{'node_id': 1010, 'host': 'kafka-jumbo1010.eqiad.wmnet', 'port': 9092, 'rack': 'E'}, {'node_id': 1013, 'host': 'kafka-jumbo1013.eqiad.wmnet', 'port': 9092, 'rack': 'F'}, {'node_id': 1007, 'host': 'kafka-jumbo1007.eqiad.wmnet', 'port': 9092, 'rack': 'C'}, {'node_id': 1016, 'host': 'kafka-jumbo1016.eqiad.wmnet', 'port': 9092, 'rack': 'E'}, {'node_id': 1015, 'host': 'kafka-jumbo1015.eqiad.wmnet', 'port': 9092, 'rack': 'F'}, {'node_id': 1009, 'host': 'kafka-jumbo1009.eqiad.wmnet', 'port': 9092, 'rack': 'D'}, {'node_id': 1012, 'host': 'kafka-jumbo1012.eqiad.wmnet', 'port': 9092, 'rack': 'E'}, {'node_id': 1014, 'host': 'kafka-jumbo1014.eqiad.wmnet', 'port': 9092, 'rack': 'F'}, {'node_id': 1008, 'host': 'kafka-jumbo1008.eqiad.wmnet', 'port': 9092, 'rack': 'D'}, {'node_id': 1011, 'host': 'kafka-jumbo1011.eqiad.wmnet', 'port': 9092, 'rack': 'E'}], 'cluster_id': 'XU3M9vwdQR2VUdeslyWHLg', 'controller_id': 1016}

Having it be exposed through spicerack would allow kafka-related cookbooks to enquire about the state and status of a cluster, eg to properly gate each step, or (in the case of T399005) ensure that the controller node is acted upon last.

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptJul 9 2025, 11:07 AM
Volans added a comment.Jul 9 2025, 1:40 PM

An immediate workaround was implemented in https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1167593 that gives us also an idea on what could be useful to expose from spicerack. If a plain KafkaAdminClient just pre-configured with the connection to the current cluster or some more advanced wrapper to extract specific information.
As a starter probably just exposing the KafkaAdminClient could be enough I guess but I'll leave to the kafka experts the decision on this.

gerritbot added a comment.Jul 17 2025, 10:40 AM

Change #1170289 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/software/spicerack@master] Kafka: expose a KafkaAdminClient builder method

https://gerrit.wikimedia.org/r/1170289

gerritbot added a project: Patch-For-Review.Jul 17 2025, 10:40 AM
brouberol changed the task status from Open to In Progress.Jul 17 2025, 10:40 AM
brouberol claimed this task.
brouberol moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.
brouberol edited projects, added: Data-Platform-SRE (2025.07.05 - 2025.07.25); removed: Data-Platform-SRE.
BTullis edited projects, added: Data-Platform-SRE (2025.07.26 - 2025.08.15); removed: Data-Platform-SRE (2025.07.05 - 2025.07.25).Jul 28 2025, 4:46 PM
BTullis moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.
bking subscribed.Jul 29 2025, 3:29 PM
brouberol closed this task as Resolved.Jul 30 2025, 1:35 PM
brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.
gerritbot added a comment.Jul 30 2025, 1:44 PM

Change #1170289 merged by jenkins-bot:

[operations/software/spicerack@master] Kafka: expose a KafkaAdminClient builder method

https://gerrit.wikimedia.org/r/1170289

Maintenance_bot removed a project: Patch-For-Review.Jul 30 2025, 2:31 PM
BTullis moved this task from Done to Reported on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.Aug 20 2025, 10:32 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits