| brouberol | |
| Wed, Jun 3, 1:00 PM |
| F86711459: Screenshot 2026-06-05 at 09.21.18.png | |
| Fri, Jun 5, 7:21 AM |
Observing kafka (listing topics, consumer groups, etc) is done using the kafka command on kafka hosts. Few people are familiar with the tooling, which slows down incident response.
We'd like to be able to observe our kafka clusters using one available off-the-self 3rd party UIs, and make it available behind CAS login.
Change #1297144 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] kubernetes/dse-k8s-eqiad: Define a kafka-ui kubeconfig
Change #1297146 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/dns@master] Define the kafka-ui internal DNS records
Change #1297148 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/dns@master] Define the kafka.w.o public record
Change #1297150 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] dse-k8s-eqiad: define the kafka-ui namespace
Change #1297146 merged by Brouberol:
[operations/dns@master] Define the kafka-ui internal DNS records
Change #1297148 merged by Brouberol:
[operations/dns@master] Define the kafka.w.o public record
Change #1297150 merged by Brouberol:
[operations/deployment-charts@master] dse-k8s: define the kafka-ui namespace
Change #1297144 merged by Brouberol:
[operations/puppet@production] kubernetes/dse-k8s: Define a kafka-ui kubeconfig
Change #1297563 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] idp: add the kafka-ui service
Change #1297564 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] trafficserver: enable access to kafka.w.o
Change #1297565 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] Define the kafka-ui chart
Change #1297566 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] Define the kafka-ui multi-cluster helmfile
Change #1297565 merged by jenkins-bot:
[operations/deployment-charts@master] Define the kafka-ui chart
Change #1297566 merged by jenkins-bot:
[operations/deployment-charts@master] Define the kafka-ui multi-cluster helmfile
Change #1297563 merged by Brouberol:
[operations/puppet@production] idp: add the kafka-ui service
Change #1297564 merged by Brouberol:
[operations/puppet@production] trafficserver: enable access to kafka.w.o
https://kafka.wikimedia.org has been deployed! I can't load the UI reliably due to some unrelated issues in drmrs having to do with piling VCLs.
@JMonton-WMF mentioned
Everything looks good on the UI, I can see and filter messages, topics, consumers... About the read-only, I'd say it's fine: - "Produce" button is disabled - Settings on topics is disabled too - "Add topic" button doesn't appear - The buttons to clean or remove topics are disabled. - "Delete offsets" on a consumer group returns 405 This cluster is in read-only mode - "Remove ACL" button is there, but it fails with "not authorized" - Changing broker properties is there in the UI, but it fails with Cluster authorization
We can't list ACLs though, because kafka-ui is connecting to kafka over port 9092 w.o any authentication.
Change #1297775 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] kafka-ui: connect to all kafka clusters
Change #1297775 merged by Brouberol:
[operations/deployment-charts@master] kafka-ui: connect to all kafka clusters
Actually, listing ACLs does work, it was just that there was none in kafka-test-eqiad.
Change #1298093 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] kafka-ui: disable latest-available version check
Change #1298093 merged by Brouberol:
[operations/deployment-charts@master] kafka-ui: disable latest-available version check
@JMeybohm pointed out that the aux clusters might be a better fit than dse for this kind of workload. I'm going to redeploy it over there.
Change #1298262 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/dns@master] dse-k8s-aux: define internal kafka-ui disc and svc records
Change #1298263 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/dns@master] Cleanup kafka-ui records pointing to the dse-k8s ingress
Change #1298266 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] aux-k8s: define the kafka-ui namespace in both clusters
Change #1298267 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] aux-k8s: define the kafka-ui helmfile and values
Change #1298264 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] aux-k8s: define the kafka-ui kubeconfigs
Change #1298268 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] dse-k8s: remove the kafka-ui namespace
Change #1298265 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] dse-k8s: remove kafka-ui kubeconfigs
Change #1298269 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] dse-k8s: remove the kafka-ui helmfile and values
Change #1298283 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] CI: add aux-k8s-codfw to the list of environments
Change #1298263 abandoned by Brouberol:
[operations/dns@master] Cleanup kafka-ui records pointing to the dse-k8s ingress
