https://julieops.readthedocs.io/en/latest/
https://github.com/kafka-ops/julie
E.g.
topics:
- name: "foo" # topicName: context.source.foo.foo
config:
replication.factor: "2"
num.partitions: "3"And much more.
Description
Description
Event Timeline
Comment Actions
I would like to see config management for Kafka topics one day, ideally integrating with EventStreamConfig (or whatever we use to declare and manage streams (and their schemas).
However, for the most part, this is only useful if we have a lot of topics that need multiple partitions, or a practice of enforcing ACLs on topics. To do the ACLs right we also need some authentication for Kafka. We can do thise now with TLS certs, but automating the process of creating and deploying TLS certs for lots and lots of different users and streams isn't really (I think?) scalable. Kafka can use Kerberos...but Kerberos is a pain with k8s so... ¯\_(ツ)_/¯
Anyway, in summary, we would like to do this, but it is still low priority.
Comment Actions
Cool, thanks for that write up @Ottomata. I am gonna switch to Stalled, to reflect noone is currently working on it but it's still a want to do.
Comment Actions
I've taken a couple of hours to whip up this PoC, very unimaginatively called kafka-configurator.
# This is what the test configuration looks like brouberol@kafka-test1006:~$ cat kafka-configurator-config.yaml cluster: bootstrap: localhost:9092 features: create_missing: config: true acls: false delete_extra: config: false acls: false topics: test-brouberol: config: retention.ms: '86200000' cleanup.policy: compact # I check what config might already exist for this topic brouberol@kafka-test1006:~$ kafka topics --describe --topic test-brouberol Topic:test-brouberol PartitionCount:1 ReplicationFactor:3 Configs:cleanup.policy=compact,delete # <-- existing config Topic: test-brouberol Partition: 0 Leader: 1008 Replicas: 1008,1006,1007 Isr: 1008,1006,1007 # I run the script that should add one config (retention.ms) and update another (cleanup.policy) brouberol@kafka-test1006:~$ ./kafka-configurator -c kafka-configurator-config.yaml 02/01/2024 13:27:55 [INFO] Applying the following configuration: {'test-brouberol': {'retention.ms': '86200000', 'cleanup.policy': 'compact'}} # I make sure the script had the expected effect brouberol@kafka-test1006:~$ kafka topics --describe --topic test-brouberol kafka-topics --zookeeper zookeeper-test1002.eqiad.wmnet/kafka/test-eqiad --describe --topic test-brouberol Topic:test-brouberol PartitionCount:1 ReplicationFactor:3 Configs:retention.ms=86200000,cleanup.policy=compact Topic: test-brouberol Partition: 0 Leader: 1008 Replicas: 1008,1006,1007 Isr: 1008,1006,1007
The script itself is very small and uses librdkafka, for forward-looking compatibility.
The way I see it, we could try to support 4 different cases:
- Adding/updating topic-level configuration when existing in the config file and not in kafka
- Deleting any topic-level configuration override existing in kafka and not in the config file
- Adding/updating ACLs when they exist in the config file and not in kafka
- Deleting any ACLs existing in kafka and not in the config file
Right now, the PoC only support scenario 1.