| JMonton-WMF | |
| Oct 14 2025, 8:24 AM |
| F70821691: Screenshot 2025-12-02 at 17.31.51.png | |
| Dec 2 2025, 4:33 PM |
| F70821692: Screenshot 2025-12-02 at 17.33.28.png | |
| Dec 2 2025, 4:33 PM |
| F70821449: Screenshot 2025-12-02 at 17.12.34.png | |
| Dec 2 2025, 4:13 PM |
| F70672099: image.png | |
| Nov 26 2025, 4:39 PM |
| F70572765: Screenshot 2025-11-24 at 12.27.30.png | |
| Nov 24 2025, 11:27 AM |
| F66749507: image.png | |
| Oct 14 2025, 8:24 AM |
| F66749501: image.png | |
| Oct 14 2025, 8:24 AM |
| F66749491: image.png | |
| Oct 14 2025, 8:24 AM |
The Kafka cluster "logging-eqiad" has a big replicas and leaders skew. 3 brokers have more than 500 replicas and 2 brokers have around 100 replicas.
It can be quickly found in Grafana:
This is happening in other clusters too.
"logging-codfw":
"main-eqiad":
"main-codfw":
"jumbo-eqiad" is fine.
The CPU of all brokers seems to be more than fine, so this doesn't look like a high priority task, but reducing the skew should make the clusters work more balanced.
This is something I've gone through over kafka-jumbo (see https://phabricator.wikimedia.org/T401599). Do reach out if you need some help!
kafka-main hosts also need to be rebooted soon-ish, is this something that should be done before, after, or it doesn't matter?
The leadership skew is expected, as it is a result of storage rebalancing. Some brokers are leaders of few large partitions, and some are leaders of many small partitions (or somewhere in between). We'll run a storage rebalancing with leadership optimization.
brouberol@kafka-logging1003:~/T407185$ topicmappr rebalance --brokers -2 --topics '.*' --storage-threshold-gb 1500 --optimize-leadership --out-file rebalancing | grep -v no-op
Topics:
...
Validating broker list:
Brokers targeted for partition offloading (< 1500.00GB storage free):
Reassignment parameters:
Ignoring partitions smaller than 512MB
Free storage mean, harmonic mean: 1515.92GB, 1462.08GB
Broker free storage limits (with a 1.00% tolerance from mean):
Sources limited to <= 1531.08GB
Destinations limited to >= 1500.76GB
Broker 1006 relocations planned:
[196.25GB] mediawiki.httpd.accesslog p3 -> 1002
[132.14GB] udp_localhost-warning p1 -> 1002
[19.40GB] k8s-mw-codfw p2 -> 1005
Broker 1004 relocations planned:
[132.14GB] udp_localhost-warning p1 -> 1001
[51.42GB] udp_localhost-err p5 -> 1001
[50.97GB] udp_localhost-err p3 -> 1005
[46.12GB] k8s-ml-serve-codfw p1 -> 1002
[19.30GB] k8s-mw-codfw p5 -> 1005
-
Total relocation volume: 647.75GB
Partition map changes:
__consumer_offsets p0: [1006 1004 1005] -> [1005 1006 1004] preferred leader
__consumer_offsets p1: [1004 1005 1006] -> [1004 1006 1005] preferred leader
__consumer_offsets p3: [1006 1005 1004] -> [1005 1006 1004] preferred leader
__consumer_offsets p4: [1004 1006 1005] -> [1005 1006 1004] preferred leader
__consumer_offsets p5: [1005 1004 1006] -> [1005 1006 1004] preferred leader
__consumer_offsets p6: [1006 1004 1005] -> [1005 1006 1004] preferred leader
__consumer_offsets p7: [1004 1005 1006] -> [1004 1006 1005] preferred leader
__consumer_offsets p11: [1005 1004 1006] -> [1005 1006 1004] preferred leader
__consumer_offsets p12: [1006 1004 1005] -> [1006 1005 1004] preferred leader
__consumer_offsets p13: [1004 1005 1006] -> [1004 1006 1005] preferred leader
__consumer_offsets p17: [1005 1004 1006] -> [1005 1006 1004] preferred leader
__consumer_offsets p18: [1006 1004 1005] -> [1006 1005 1004] preferred leader
__consumer_offsets p19: [1004 1005 1006] -> [1004 1006 1005] preferred leader
__consumer_offsets p23: [1005 1004 1006] -> [1005 1006 1004] preferred leader
__consumer_offsets p24: [1006 1004 1005] -> [1006 1005 1004] preferred leader
__consumer_offsets p25: [1004 1005 1006] -> [1004 1006 1005] preferred leader
__consumer_offsets p29: [1005 1004 1006] -> [1005 1006 1004] preferred leader
__consumer_offsets p30: [1006 1004 1005] -> [1006 1005 1004] preferred leader
__consumer_offsets p31: [1004 1005 1006] -> [1004 1006 1005] preferred leader
__consumer_offsets p35: [1005 1004 1006] -> [1005 1006 1004] preferred leader
__consumer_offsets p36: [1006 1004 1005] -> [1006 1005 1004] preferred leader
__consumer_offsets p37: [1004 1005 1006] -> [1004 1006 1005] preferred leader
__consumer_offsets p41: [1005 1004 1006] -> [1005 1006 1004] preferred leader
__consumer_offsets p42: [1006 1004 1005] -> [1006 1005 1004] preferred leader
__consumer_offsets p43: [1004 1005 1006] -> [1004 1006 1005] preferred leader
__consumer_offsets p47: [1005 1004 1006] -> [1005 1006 1004] preferred leader
__consumer_offsets p48: [1006 1004 1005] -> [1006 1005 1004] preferred leader
__consumer_offsets p49: [1004 1005 1006] -> [1004 1006 1005] preferred leader
codfw.mediawiki.client.error p0: [1004 1005 1006] -> [1005 1006 1004] preferred leader
codfw.mediawiki.client.error p1: [1005 1006 1004] -> [1006 1005 1004] preferred leader
codfw.mediawiki.client.error p2: [1006 1004 1005] -> [1004 1006 1005] preferred leader
codfw.mediawiki.client.error p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
codfw.w3c.reportingapi.network_error p0: [1005 1004 1006] -> [1004 1006 1005] preferred leader
codfw.w3c.reportingapi.network_error p1: [1006 1005 1004] -> [1005 1006 1004] preferred leader
codfw.w3c.reportingapi.network_error p2: [1004 1006 1005] -> [1006 1005 1004] preferred leader
codfw.w3c.reportingapi.network_error p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
codfw.w3c.reportingapi.network_error p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
deprecated p1: [1006 1004 1005] -> [1006 1005 1004] preferred leader
deprecated p2: [1004 1005 1006] -> [1004 1006 1005] preferred leader
deprecated p3: [1005 1004 1006] -> [1005 1006 1004] preferred leader
eqiad.client.error p1: [1006 1004 1005] -> [1006 1005 1004] preferred leader
eqiad.client.error p2: [1004 1005 1006] -> [1004 1006 1005] preferred leader
eqiad.client.error p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
eqiad.client.error p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
eqiad.client.error p6: [1005 1004 1006] -> [1005 1006 1004] preferred leader
eqiad.client.error p10: [1006 1004 1005] -> [1006 1005 1004] preferred leader
eqiad.client.error p11: [1004 1005 1006] -> [1004 1006 1005] preferred leader
eqiad.eventgate-logging-external.error.validation p1: [1006 1004 1005] -> [1006 1005 1004] preferred leader
eqiad.eventgate-logging-external.error.validation p2: [1004 1005 1006] -> [1004 1006 1005] preferred leader
eqiad.eventgate-logging-external.error.validation p3: [1005 1004 1006] -> [1005 1006 1004] preferred leader
eqiad.eventgate-logging-external.test.event p0: [1004 1006 1005] -> [1005 1006 1004] preferred leader
eqiad.eventgate-logging-external.test.event p1: [1005 1004 1006] -> [1006 1005 1004] preferred leader
eqiad.eventgate-logging-external.test.event p2: [1006 1005 1004] -> [1004 1006 1005] preferred leader
eqiad.eventgate-logging-external.test.event p3: [1004 1005 1006] -> [1004 1006 1005] preferred leader
eqiad.eventgate-logging-external.test.event p5: [1006 1004 1005] -> [1006 1005 1004] preferred leader
eqiad.kaios_app.error p0: [1005 1004 1006] -> [1004 1006 1005] preferred leader
eqiad.kaios_app.error p1: [1006 1005 1004] -> [1005 1006 1004] preferred leader
eqiad.kaios_app.error p2: [1004 1006 1005] -> [1006 1005 1004] preferred leader
eqiad.kaios_app.error p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
eqiad.kaios_app.error p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
eqiad.mediawiki.client.error p0: [1004 1005 1006] -> [1005 1006 1004] preferred leader
eqiad.mediawiki.client.error p1: [1005 1006 1004] -> [1006 1005 1004] preferred leader
eqiad.mediawiki.client.error p2: [1006 1004 1005] -> [1004 1006 1005] preferred leader
eqiad.mediawiki.client.error p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
eqiad.test.event p0: [1005 1006 1004] -> [1004 1006 1005] preferred leader
eqiad.test.event p1: [1006 1004 1005] -> [1005 1006 1004] preferred leader
eqiad.test.event p2: [1004 1005 1006] -> [1006 1005 1004] preferred leader
eqiad.test.event p3: [1005 1004 1006] -> [1005 1006 1004] preferred leader
eqiad.w3c.reportingapi.network_error p0: [1004 1005 1006] -> [1005 1006 1004] preferred leader
eqiad.w3c.reportingapi.network_error p1: [1005 1006 1004] -> [1006 1005 1004] preferred leader
eqiad.w3c.reportingapi.network_error p2: [1006 1004 1005] -> [1004 1006 1005] preferred leader
eqiad.w3c.reportingapi.network_error p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
k8s-aux-k8s-codfw p0: [1001 1002 1006] -> [1002 1006 1001] preferred leader
k8s-aux-k8s-codfw p1: [1002 1006 1004] -> [1002 1004 1006] preferred leader
k8s-aux-k8s-codfw p3: [1004 1005 1001] -> [1005 1004 1001] preferred leader
k8s-aux-k8s-codfw p4: [1005 1001 1002] -> [1001 1005 1002] preferred leader
k8s-aux-k8s-codfw p5: [1001 1006 1004] -> [1001 1004 1006] preferred leader
k8s-aux-k8s-eqiad p2: [1005 1001 1002] -> [1001 1005 1002] preferred leader
k8s-aux-k8s-eqiad p3: [1001 1002 1006] -> [1002 1006 1001] preferred leader
k8s-aux-k8s-eqiad p4: [1002 1006 1004] -> [1006 1004 1002] preferred leader
k8s-aux-k8s-eqiad p5: [1006 1005 1001] -> [1001 1005 1006] preferred leader
k8s-codfw p0: [1006 1005 1001] -> [1005 1006 1001] preferred leader
k8s-codfw p1: [1004 1001 1002] -> [1001 1004 1002] preferred leader
k8s-codfw p2: [1005 1002 1006] -> [1005 1006 1002] preferred leader
k8s-codfw p3: [1001 1006 1004] -> [1004 1006 1001] preferred leader
k8s-codfw p5: [1006 1001 1002] -> [1006 1002 1001] preferred leader
k8s-dse-k8s-codfw p0: [1004 1002 1006] -> [1004 1006 1002] preferred leader
k8s-dse-k8s-codfw p1: [1005 1006 1004] -> [1004 1006 1005] preferred leader
k8s-dse-k8s-codfw p2: [1001 1004 1005] -> [1005 1004 1001] preferred leader
k8s-dse-k8s-codfw p4: [1006 1001 1002] -> [1001 1006 1002] preferred leader
k8s-dse-k8s-codfw p5: [1004 1006 1005] -> [1006 1005 1004] preferred leader
k8s-dse-k8s-eqiad p0: [1006 1005 1001] -> [1001 1006 1005] preferred leader
k8s-dse-k8s-eqiad p1: [1004 1001 1002] -> [1004 1002 1001] preferred leader
k8s-dse-k8s-eqiad p2: [1005 1002 1006] -> [1002 1006 1005] preferred leader
k8s-dse-k8s-eqiad p3: [1001 1006 1004] -> [1006 1004 1001] preferred leader
k8s-dse-k8s-eqiad p4: [1002 1004 1005] -> [1005 1004 1002] preferred leader
k8s-dse-k8s-eqiad p5: [1006 1001 1002] -> [1006 1002 1001] preferred leader
k8s-eqiad p0: [1004 1001 1002] -> [1004 1002 1001] preferred leader
k8s-eqiad p1: [1005 1002 1006] -> [1002 1006 1005] preferred leader
k8s-eqiad p2: [1001 1006 1004] -> [1006 1004 1001] preferred leader
k8s-eqiad p3: [1002 1004 1005] -> [1005 1004 1002] preferred leader
k8s-eqiad p4: [1006 1005 1001] -> [1001 1006 1005] preferred leader
k8s-eqiad p5: [1004 1002 1006] -> [1004 1006 1002] preferred leader
k8s-ml-serve-codfw p0: [1006 1004 1005] -> [1005 1004 1006] preferred leader
k8s-ml-serve-codfw p1: [1004 1005 1001] -> [1005 1002 1001] replaced broker
k8s-ml-serve-codfw p2: [1005 1001 1002] -> [1002 1005 1001] preferred leader
k8s-ml-serve-codfw p3: [1001 1002 1006] -> [1001 1006 1002] preferred leader
k8s-ml-serve-eqiad p2: [1005 1001 1002] -> [1001 1005 1002] preferred leader
k8s-ml-serve-eqiad p3: [1001 1002 1006] -> [1006 1002 1001] preferred leader
k8s-ml-serve-eqiad p4: [1002 1006 1004] -> [1002 1004 1006] preferred leader
k8s-ml-serve-eqiad p5: [1006 1005 1001] -> [1005 1006 1001] preferred leader
k8s-ml-staging-codfw p0: [1001 1002 1006] -> [1006 1002 1001] preferred leader
k8s-ml-staging-codfw p1: [1002 1006 1004] -> [1004 1006 1002] preferred leader
k8s-ml-staging-codfw p2: [1006 1004 1005] -> [1006 1005 1004] preferred leader
k8s-ml-staging-codfw p3: [1004 1005 1001] -> [1001 1005 1004] preferred leader
k8s-ml-staging-codfw p4: [1005 1001 1002] -> [1002 1005 1001] preferred leader
k8s-ml-staging-codfw p5: [1001 1006 1004] -> [1001 1004 1006] preferred leader
k8s-mw-codfw p0: [1004 1001 1002] -> [1004 1002 1001] preferred leader
k8s-mw-codfw p1: [1005 1002 1006] -> [1002 1006 1005] preferred leader
k8s-mw-codfw p2: [1001 1006 1004] -> [1005 1004 1001] replaced broker
k8s-mw-codfw p3: [1002 1004 1005] -> [1004 1002 1005] preferred leader
k8s-mw-codfw p4: [1006 1005 1001] -> [1001 1006 1005] preferred leader
k8s-mw-codfw p5: [1004 1002 1006] -> [1006 1005 1002] replaced broker
k8s-mw-eqiad p2: [1002 1001 1006] -> [1002 1006 1001] preferred leader
k8s-mw-eqiad p4: [1004 1006 1005] -> [1006 1005 1004] preferred leader
k8s-mw-eqiad p5: [1005 1001 1002] -> [1005 1002 1001] preferred leader
k8s-staging-codfw p0: [1006 1004 1005] -> [1005 1006 1004] preferred leader
k8s-staging-codfw p2: [1005 1001 1002] -> [1002 1001 1005] preferred leader
k8s-staging-codfw p3: [1001 1002 1006] -> [1001 1006 1002] preferred leader
k8s-staging-codfw p4: [1002 1006 1004] -> [1006 1004 1002] preferred leader
k8s-staging-codfw p5: [1006 1005 1001] -> [1005 1006 1001] preferred leader
k8s-staging-eqiad p0: [1001 1006 1004] -> [1004 1006 1001] preferred leader
k8s-staging-eqiad p1: [1002 1004 1005] -> [1005 1004 1002] preferred leader
k8s-staging-eqiad p2: [1006 1005 1001] -> [1001 1006 1005] preferred leader
k8s-staging-eqiad p3: [1004 1001 1002] -> [1004 1002 1001] preferred leader
k8s-staging-eqiad p4: [1005 1002 1006] -> [1002 1006 1005] preferred leader
k8s-staging-eqiad p5: [1001 1004 1005] -> [1005 1004 1001] preferred leader
logback- p0: [1001 1005 1002] -> [1002 1001 1005] preferred leader
logback- p1: [1002 1001 1006] -> [1001 1006 1002] preferred leader
logback- p2: [1006 1002 1004] -> [1006 1004 1002] preferred leader
logback- p3: [1004 1006 1005] -> [1006 1005 1004] preferred leader
logback- p4: [1005 1004 1001] -> [1001 1004 1005] preferred leader
logback- p5: [1001 1002 1006] -> [1001 1006 1002] preferred leader
logback-3 p0: [1004 1005 1006] -> [1004 1006 1005] preferred leader
logback-3 p1: [1005 1006 1004] -> [1006 1005 1004] preferred leader
logback-3 p2: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-3 p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
logback-3 p5: [1006 1005 1004] -> [1004 1006 1005] preferred leader
logback-4 p0: [1006 1004 1005] -> [1005 1006 1004] preferred leader
logback-4 p1: [1004 1005 1006] -> [1005 1006 1004] preferred leader
logback-4 p2: [1005 1006 1004] -> [1006 1005 1004] preferred leader
logback-4 p5: [1005 1004 1006] -> [1005 1006 1004] preferred leader
logback-6 p0: [1004 1005 1006] -> [1006 1005 1004] preferred leader
logback-6 p1: [1005 1006 1004] -> [1004 1006 1005] preferred leader
logback-6 p2: [1006 1004 1005] -> [1004 1006 1005] preferred leader
logback-6 p3: [1004 1006 1005] -> [1005 1006 1004] preferred leader
logback-6 p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
logback-7 p0: [1005 1006 1004] -> [1004 1006 1005] preferred leader
logback-7 p1: [1006 1004 1005] -> [1005 1006 1004] preferred leader
logback-7 p2: [1004 1005 1006] -> [1006 1005 1004] preferred leader
logback-7 p3: [1005 1004 1006] -> [1005 1006 1004] preferred leader
logback-critical p0: [1004 1005 1006] -> [1005 1006 1004] preferred leader
logback-critical p1: [1005 1006 1004] -> [1006 1005 1004] preferred leader
logback-critical p2: [1006 1004 1005] -> [1004 1006 1005] preferred leader
logback-critical p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
logback-debug p0: [1001 1006 1004] -> [1004 1006 1001] preferred leader
logback-debug p2: [1006 1005 1001] -> [1005 1006 1001] preferred leader
logback-debug p3: [1004 1001 1002] -> [1001 1004 1002] preferred leader
logback-debug p4: [1005 1002 1006] -> [1006 1005 1002] preferred leader
logback-error p2: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-error p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
logback-error p6: [1004 1005 1006] -> [1004 1006 1005] preferred leader
logback-error p8: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-error p10: [1005 1004 1006] -> [1005 1006 1004] preferred leader
logback-info p0: [1005 1006 1004] -> [1004 1006 1005] preferred leader
logback-info p1: [1006 1004 1005] -> [1005 1006 1004] preferred leader
logback-info p2: [1004 1005 1006] -> [1006 1005 1004] preferred leader
logback-info p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-info p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
logback-info p6: [1005 1004 1006] -> [1005 1006 1004] preferred leader
logback-info p10: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-info p11: [1004 1005 1006] -> [1004 1006 1005] preferred leader
logback-warn p1: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-warn p2: [1004 1005 1006] -> [1004 1006 1005] preferred leader
logback-warn p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-warn p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
logback-warn p6: [1005 1004 1006] -> [1005 1006 1004] preferred leader
logback-warn p10: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-warn p11: [1004 1005 1006] -> [1004 1006 1005] preferred leader
logback-warning p1: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-warning p2: [1004 1005 1006] -> [1004 1006 1005] preferred leader
logback-warning p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-warning p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
logback-warning p6: [1005 1004 1006] -> [1005 1006 1004] preferred leader
logback-warning p10: [1006 1004 1005] -> [1006 1005 1004] preferred leader
logback-warning p11: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mediawiki.httpd.accesslog p0: [1004 1005 1001] -> [1005 1004 1001] preferred leader
mediawiki.httpd.accesslog p1: [1001 1002 1006] -> [1002 1006 1001] preferred leader
mediawiki.httpd.accesslog p2: [1002 1001 1004] -> [1001 1004 1002] preferred leader
mediawiki.httpd.accesslog p3: [1005 1006 1004] -> [1004 1005 1002] replaced broker
mediawiki.httpd.accesslog p4: [1006 1002 1005] -> [1005 1006 1002] preferred leader
mediawiki.httpd.accesslog p5: [1004 1002 1006] -> [1002 1004 1006] preferred leader
mediawiki.httpd.accesslog p6: [1001 1005 1006] -> [1006 1005 1001] preferred leader
mediawiki.httpd.accesslog p7: [1002 1001 1004] -> [1001 1004 1002] preferred leader
mediawiki.httpd.accesslog p8: [1005 1001 1004] -> [1004 1005 1001] preferred leader
mediawiki.httpd.accesslog p9: [1006 1001 1002] -> [1001 1006 1002] preferred leader
mediawiki.httpd.accesslog p10: [1002 1004 1006] -> [1006 1004 1002] preferred leader
mediawiki.httpd.accesslog p11: [1004 1001 1005] -> [1004 1005 1001] preferred leader
mediawiki.httpd.accesslog-sampled p0: [1006 1001 1005] -> [1006 1005 1001] preferred leader
mediawiki.httpd.accesslog-sampled p2: [1005 1001 1002] -> [1002 1005 1001] preferred leader
mediawiki.httpd.accesslog-sampled p3: [1001 1002 1006] -> [1006 1002 1001] preferred leader
mediawiki.php-fpm.slowlog p0: [1006 1005 1004] -> [1005 1004 1006] preferred leader
mediawiki.php-fpm.slowlog p1: [1004 1006 1005] -> [1005 1006 1004] preferred leader
mediawiki.php-fpm.slowlog p2: [1005 1004 1006] -> [1004 1006 1005] preferred leader
mediawiki.php-fpm.slowlog p3: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mediawiki.php-fpm.slowlog p4: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mediawiki.php-fpm.slowlog p5: [1005 1006 1004] -> [1004 1006 1005] preferred leader
mm2-configs.target.internal p0: [1006 1002 1004] -> [1006 1004 1002] preferred leader
mm2-offsets.target.internal p0: [1006 1005 1001] -> [1001 1006 1005] preferred leader
mm2-offsets.target.internal p1: [1004 1001 1002] -> [1002 1004 1001] preferred leader
mm2-offsets.target.internal p2: [1005 1002 1006] -> [1005 1006 1002] preferred leader
mm2-offsets.target.internal p3: [1001 1006 1004] -> [1006 1004 1001] preferred leader
mm2-offsets.target.internal p4: [1002 1004 1005] -> [1004 1005 1002] preferred leader
mm2-offsets.target.internal p5: [1006 1001 1002] -> [1006 1002 1001] preferred leader
mm2-offsets.target.internal p6: [1004 1002 1006] -> [1006 1004 1002] preferred leader
mm2-offsets.target.internal p7: [1005 1006 1004] -> [1004 1006 1005] preferred leader
mm2-offsets.target.internal p8: [1001 1004 1005] -> [1005 1004 1001] preferred leader
mm2-offsets.target.internal p10: [1006 1002 1004] -> [1006 1004 1002] preferred leader
mm2-offsets.target.internal p11: [1004 1006 1005] -> [1006 1005 1004] preferred leader
mm2-offsets.target.internal p12: [1005 1004 1001] -> [1001 1005 1004] preferred leader
mm2-offsets.target.internal p14: [1002 1001 1006] -> [1002 1006 1001] preferred leader
mm2-offsets.target.internal p15: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mm2-offsets.target.internal p16: [1004 1005 1001] -> [1005 1004 1001] preferred leader
mm2-offsets.target.internal p17: [1005 1001 1002] -> [1001 1005 1002] preferred leader
mm2-offsets.target.internal p18: [1001 1002 1006] -> [1006 1002 1001] preferred leader
mm2-offsets.target.internal p19: [1002 1006 1004] -> [1002 1004 1006] preferred leader
mm2-offsets.target.internal p20: [1006 1005 1001] -> [1005 1006 1001] preferred leader
mm2-offsets.target.internal p21: [1004 1001 1002] -> [1004 1002 1001] preferred leader
mm2-offsets.target.internal p22: [1005 1002 1006] -> [1005 1006 1002] preferred leader
mm2-offsets.target.internal p23: [1001 1006 1004] -> [1001 1004 1006] preferred leader
mm2-status.target.internal p0: [1006 1002 1004] -> [1004 1006 1002] preferred leader
mm2-status.target.internal p1: [1004 1006 1005] -> [1004 1005 1006] preferred leader
mm2-status.target.internal p3: [1001 1005 1002] -> [1005 1002 1001] preferred leader
mm2-status.target.internal p4: [1002 1001 1006] -> [1002 1006 1001] preferred leader
mwlog-debug p0: [1005 1006 1004] -> [1005 1004 1006] preferred leader
mwlog-debug p1: [1006 1004 1005] -> [1004 1006 1005] preferred leader
mwlog-debug p2: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-debug p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mwlog-debug p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-debug p6: [1005 1004 1006] -> [1005 1006 1004] preferred leader
mwlog-debug p10: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mwlog-debug p11: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-err p0: [1004 1005 1006] -> [1005 1006 1004] preferred leader
mwlog-err p1: [1005 1006 1004] -> [1006 1005 1004] preferred leader
mwlog-err p2: [1006 1004 1005] -> [1004 1006 1005] preferred leader
mwlog-err p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
mwlog-err p6: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-err p8: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mwlog-err p10: [1005 1004 1006] -> [1005 1006 1004] preferred leader
mwlog-error p0: [1006 1005 1004] -> [1004 1006 1005] preferred leader
mwlog-error p1: [1004 1006 1005] -> [1005 1006 1004] preferred leader
mwlog-error p2: [1005 1004 1006] -> [1006 1005 1004] preferred leader
mwlog-error p3: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mwlog-error p4: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-info p0: [1005 1006 1004] -> [1006 1005 1004] preferred leader
mwlog-info p1: [1006 1004 1005] -> [1004 1006 1005] preferred leader
mwlog-info p2: [1004 1005 1006] -> [1005 1006 1004] preferred leader
mwlog-info p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mwlog-info p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-info p6: [1005 1004 1006] -> [1005 1006 1004] preferred leader
mwlog-info p10: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mwlog-info p11: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-notice p1: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mwlog-notice p2: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-notice p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mwlog-notice p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-notice p6: [1005 1004 1006] -> [1005 1006 1004] preferred leader
mwlog-notice p10: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mwlog-notice p11: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-warning p0: [1004 1005 1006] -> [1005 1006 1004] preferred leader
mwlog-warning p1: [1005 1006 1004] -> [1006 1005 1004] preferred leader
mwlog-warning p2: [1006 1004 1005] -> [1004 1006 1005] preferred leader
mwlog-warning p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
mwlog-warning p6: [1004 1005 1006] -> [1004 1006 1005] preferred leader
mwlog-warning p8: [1006 1004 1005] -> [1006 1005 1004] preferred leader
mwlog-warning p10: [1005 1004 1006] -> [1005 1006 1004] preferred leader
rsyslog p1: [1005 1004 1006] -> [1005 1006 1004] preferred leader
rsyslog p3: [1004 1005 1006] -> [1004 1006 1005] preferred leader
rsyslog p5: [1006 1004 1005] -> [1006 1005 1004] preferred leader
rsyslog-alert p1: [1005 1004 1006] -> [1005 1006 1004] preferred leader
rsyslog-alert p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
rsyslog-crit p0: [1004 1005 1006] -> [1004 1006 1005] preferred leader
rsyslog-crit p2: [1006 1004 1005] -> [1006 1005 1004] preferred leader
rsyslog-crit p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
rsyslog-debug p0: [1004 1005 1006] -> [1004 1006 1005] preferred leader
rsyslog-debug p2: [1006 1004 1005] -> [1006 1005 1004] preferred leader
rsyslog-debug p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
rsyslog-emerg p0: [1004 1005 1006] -> [1004 1006 1005] preferred leader
rsyslog-emerg p2: [1006 1004 1005] -> [1006 1005 1004] preferred leader
rsyslog-emerg p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
rsyslog-er p0: [1004 1005 1006] -> [1004 1006 1005] preferred leader
rsyslog-er p2: [1006 1004 1005] -> [1006 1005 1004] preferred leader
rsyslog-er p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
rsyslog-err p2: [1006 1004 1005] -> [1006 1005 1004] preferred leader
rsyslog-err p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
rsyslog-error p0: [1004 1005 1006] -> [1004 1006 1005] preferred leader
rsyslog-error p2: [1006 1004 1005] -> [1006 1005 1004] preferred leader
rsyslog-error p4: [1005 1004 1006] -> [1005 1006 1004] preferred leader
rsyslog-info p0: [1006 1004 1005] -> [1004 1006 1005] preferred leader
rsyslog-info p1: [1004 1001 1006] -> [1001 1006 1004] preferred leader
rsyslog-info p2: [1005 1006 1001] -> [1001 1005 1006] preferred leader
rsyslog-info p4: [1004 1006 1005] -> [1006 1005 1004] preferred leader
rsyslog-info p5: [1005 1004 1006] -> [1006 1005 1004] preferred leader
rsyslog-notice p0: [1006 1004 1005] -> [1005 1004 1006] preferred leader
rsyslog-notice p1: [1004 1005 1001] -> [1005 1004 1001] preferred leader
rsyslog-notice p2: [1005 1006 1004] -> [1004 1005 1006] preferred leader
rsyslog-notice p3: [1006 1005 1004] -> [1005 1004 1006] preferred leader
rsyslog-notice p4: [1004 1006 1005] -> [1006 1004 1005] preferred leader
rsyslog-notice p5: [1005 1004 1006] -> [1004 1005 1006] preferred leader
rsyslog-warn p0: [1006 1004 1005] -> [1006 1005 1004] preferred leader
rsyslog-warn p1: [1004 1005 1006] -> [1005 1004 1006] preferred leader
rsyslog-warn p2: [1005 1006 1004] -> [1004 1005 1006] preferred leader
rsyslog-warn p4: [1004 1006 1005] -> [1005 1004 1006] preferred leader
rsyslog-warn p5: [1005 1004 1006] -> [1004 1005 1006] preferred leader
rsyslog-warning p0: [1006 1004 1005] -> [1006 1005 1004] preferred leader
rsyslog-warning p1: [1004 1005 1006] -> [1005 1004 1006] preferred leader
rsyslog-warning p2: [1005 1006 1004] -> [1004 1005 1006] preferred leader
rsyslog-warning p4: [1004 1006 1005] -> [1005 1004 1006] preferred leader
rsyslog-warning p5: [1001 1004 1006] -> [1004 1006 1001] preferred leader
staging.eventgate-logging-external.error.validation p0: [1006 1004 1005] -> [1006 1005 1004] preferred leader
staging.eventgate-logging-external.error.validation p2: [1005 1006 1004] -> [1006 1005 1004] preferred leader
staging.eventgate-logging-external.error.validation p4: [1004 1006 1005] -> [1005 1004 1006] preferred leader
staging.eventgate-logging-external.error.validation p5: [1005 1004 1006] -> [1004 1005 1006] preferred leader
staging.eventgate-logging-external.test.event p0: [1005 1006 1004] -> [1006 1005 1004] preferred leader
staging.eventgate-logging-external.test.event p1: [1006 1004 1005] -> [1005 1004 1006] preferred leader
staging.eventgate-logging-external.test.event p2: [1004 1005 1006] -> [1005 1004 1006] preferred leader
staging.eventgate-logging-external.test.event p3: [1005 1004 1006] -> [1004 1005 1006] preferred leader
staging.eventgate-logging-external.test.event p5: [1004 1006 1005] -> [1005 1004 1006] preferred leader
staging.mediawiki.client.error p1: [1005 1006 1004] -> [1006 1005 1004] preferred leader
staging.mediawiki.client.error p2: [1006 1004 1005] -> [1004 1005 1006] preferred leader
staging.mediawiki.client.error p3: [1004 1006 1005] -> [1005 1004 1006] preferred leader
staging.mediawiki.client.error p4: [1005 1004 1006] -> [1004 1005 1006] preferred leader
staging.test.event p1: [1006 1005 1004] -> [1004 1005 1006] preferred leader
staging.test.event p2: [1004 1006 1005] -> [1006 1005 1004] preferred leader
staging.test.event p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
staging.test.event p5: [1004 1005 1006] -> [1005 1004 1006] preferred leader
staging.w3c.reportingapi.network_error p0: [1006 1005 1004] -> [1004 1005 1006] preferred leader
staging.w3c.reportingapi.network_error p1: [1004 1006 1005] -> [1006 1005 1004] preferred leader
staging.w3c.reportingapi.network_error p2: [1005 1004 1006] -> [1004 1005 1006] preferred leader
staging.w3c.reportingapi.network_error p3: [1006 1004 1005] -> [1006 1005 1004] preferred leader
staging.w3c.reportingapi.network_error p4: [1004 1005 1006] -> [1005 1004 1006] preferred leader
staging.w3c.reportingapi.network_error p5: [1005 1006 1004] -> [1004 1005 1006] preferred leader
udp_127.0.0.1-debug p0: [1004 1006 1005] -> [1006 1005 1004] preferred leader
udp_127.0.0.1-debug p1: [1005 1004 1006] -> [1004 1005 1006] preferred leader
udp_127.0.0.1-debug p2: [1006 1005 1004] -> [1005 1004 1006] preferred leader
udp_127.0.0.1-debug p3: [1004 1005 1006] -> [1005 1004 1006] preferred leader
udp_127.0.0.1-debug p4: [1005 1006 1004] -> [1004 1005 1006] preferred leader
udp_127.0.0.1-debug p5: [1006 1004 1005] -> [1006 1005 1004] preferred leader
udp_127.0.0.1-info p0: [1005 1006 1004] -> [1005 1004 1006] preferred leader
udp_127.0.0.1-info p1: [1006 1004 1005] -> [1004 1005 1006] preferred leader
udp_127.0.0.1-info p2: [1004 1005 1006] -> [1006 1005 1004] preferred leader
udp_127.0.0.1-info p3: [1005 1004 1006] -> [1005 1006 1004] preferred leader
udp_127.0.0.1-info p5: [1004 1006 1005] -> [1005 1004 1006] preferred leader
udp_json_logback_compat_topic p0: [1006 1004 1005] -> [1004 1005 1006] preferred leader
udp_json_logback_compat_topic p1: [1004 1005 1006] -> [1006 1005 1004] preferred leader
udp_json_logback_compat_topic p2: [1005 1006 1004] -> [1004 1005 1006] preferred leader
udp_json_logback_compat_topic p4: [1004 1006 1005] -> [1005 1004 1006] preferred leader
udp_json_logback_compat_topic p5: [1005 1004 1006] -> [1004 1005 1006] preferred leader
udp_json_logback_compat_topic_info p0: [1005 1004 1006] -> [1006 1005 1004] preferred leader
udp_json_logback_compat_topic_info p1: [1006 1005 1004] -> [1005 1004 1006] preferred leader
udp_json_logback_compat_topic_info p2: [1004 1006 1005] -> [1005 1004 1006] preferred leader
udp_json_logback_compat_topic_info p3: [1005 1006 1004] -> [1004 1005 1006] preferred leader
udp_json_logback_compat_topic_info p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
udp_json_logback_compat_topic_info p5: [1004 1005 1006] -> [1005 1004 1006] preferred leader
udp_localhost-crit p0: [1002 1004 1005] -> [1004 1005 1002] preferred leader
udp_localhost-crit p1: [1006 1005 1001] -> [1005 1006 1001] preferred leader
udp_localhost-crit p2: [1004 1001 1002] -> [1004 1002 1001] preferred leader
udp_localhost-crit p3: [1005 1002 1006] -> [1002 1006 1005] preferred leader
udp_localhost-crit p4: [1001 1006 1004] -> [1006 1004 1001] preferred leader
udp_localhost-crit p5: [1002 1005 1001] -> [1002 1001 1005] preferred leader
udp_localhost-debug p0: [1006 1004 1005] -> [1005 1006 1004] preferred leader
udp_localhost-debug p1: [1004 1005 1006] -> [1006 1005 1004] preferred leader
udp_localhost-debug p2: [1005 1006 1004] -> [1004 1006 1005] preferred leader
udp_localhost-debug p3: [1006 1005 1004] -> [1004 1006 1005] preferred leader
udp_localhost-debug p5: [1005 1004 1006] -> [1005 1006 1004] preferred leader
udp_localhost-emerg p0: [1005 1006 1004] -> [1006 1005 1004] preferred leader
udp_localhost-emerg p1: [1006 1004 1005] -> [1004 1006 1005] preferred leader
udp_localhost-emerg p2: [1004 1005 1006] -> [1005 1006 1004] preferred leader
udp_localhost-emerg p4: [1006 1004 1005] -> [1006 1005 1004] preferred leader
udp_localhost-emerg p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
udp_localhost-err p3: [1001 1006 1004] -> [1001 1005 1006] replaced broker
udp_localhost-err p4: [1006 1004 1005] -> [1005 1006 1004] preferred leader
udp_localhost-err p5: [1004 1005 1006] -> [1001 1006 1005] replaced broker
udp_localhost-info p0: [1004 1005 1006] -> [1006 1005 1004] preferred leader
udp_localhost-info p1: [1005 1006 1004] -> [1006 1004 1005] preferred leader
udp_localhost-info p3: [1004 1001 1002] -> [1004 1002 1001] preferred leader
udp_localhost-info p4: [1005 1004 1006] -> [1004 1006 1005] preferred leader
udp_localhost-info p5: [1006 1005 1001] -> [1001 1005 1006] preferred leader
udp_localhost-info-0 p0: [1004 1005 1006] -> [1006 1005 1004] preferred leader
udp_localhost-info-0 p1: [1005 1006 1004] -> [1005 1004 1006] preferred leader
udp_localhost-info-0 p3: [1004 1006 1005] -> [1004 1005 1006] preferred leader
udp_localhost-info-0 p5: [1006 1005 1004] -> [1006 1004 1005] preferred leader
udp_localhost-info-1 p0: [1006 1004 1005] -> [1005 1004 1006] preferred leader
udp_localhost-info-1 p1: [1004 1005 1006] -> [1005 1004 1006] preferred leader
udp_localhost-info-1 p2: [1005 1006 1004] -> [1006 1004 1005] preferred leader
udp_localhost-info-1 p3: [1006 1005 1004] -> [1006 1004 1005] preferred leader
udp_localhost-info-2 p0: [1005 1004 1006] -> [1006 1004 1005] preferred leader
udp_localhost-info-2 p1: [1006 1005 1004] -> [1004 1006 1005] preferred leader
udp_localhost-info-2 p2: [1004 1006 1005] -> [1005 1004 1006] preferred leader
udp_localhost-info-2 p3: [1005 1006 1004] -> [1005 1004 1006] preferred leader
udp_localhost-info-2 p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
udp_localhost-notice p0: [1004 1005 1006] -> [1004 1006 1005] preferred leader
udp_localhost-notice p1: [1005 1006 1004] -> [1006 1004 1005] preferred leader
udp_localhost-notice p2: [1006 1004 1005] -> [1004 1006 1005] preferred leader
udp_localhost-notice p3: [1004 1006 1005] -> [1004 1005 1006] preferred leader
udp_localhost-notice p5: [1006 1005 1004] -> [1006 1004 1005] preferred leader
udp_localhost-warn p1: [1006 1005 1004] -> [1005 1004 1006] preferred leader
udp_localhost-warn p2: [1004 1006 1005] -> [1006 1004 1005] preferred leader
udp_localhost-warn p3: [1005 1006 1004] -> [1005 1004 1006] preferred leader
udp_localhost-warn p5: [1004 1005 1006] -> [1004 1006 1005] preferred leader
udp_localhost-warning p0: [1005 1002 1006] -> [1006 1005 1002] preferred leader
udp_localhost-warning p1: [1006 1005 1004] -> [1002 1005 1001] replaced broker
udp_localhost-warning p2: [1004 1006 1005] -> [1004 1005 1006] preferred leader
udp_localhost-warning p3: [1005 1006 1004] -> [1004 1005 1006] preferred leader
udp_localhost-warning p4: [1002 1004 1001] -> [1001 1004 1002] preferred leader
Broker distribution:
degree [min/max/avg]: 4/4/4.00 -> 4/4/4.00
-
Broker 1001 - leader: 35, follower: 68, total: 103
Broker 1002 - leader: 31, follower: 61, total: 92
Broker 1004 - leader: 176, follower: 352, total: 528
Broker 1005 - leader: 177, follower: 357, total: 534
Broker 1006 - leader: 180, follower: 356, total: 536
Storage free change estimations:
range: 720.40GB -> 17.02GB
range spread: 61.65% -> 1.13%
std. deviation: 281.14GB -> 6.19GB
min-max: 1168.49GB, 1888.89GB -> 1510.50GB, 1527.52GB
-
Broker 1001: 1694.49 -> 1510.93 (-183.56GB, -10.83%)
Broker 1002: 1888.89 -> 1514.38 (-374.51GB, -19.83%)
Broker 1004: 1210.55 -> 1510.50 (+299.95GB, 24.78%)
Broker 1005: 1617.20 -> 1527.52 (-89.68GB, -5.55%)
Broker 1006: 1168.49 -> 1516.28 (+347.79GB, 29.76%)I'll probably wait until https://phabricator.wikimedia.org/T409800 is figured out first.
The kafka-main rebalance question is now pretty critical to figure out. One of the broker's certificates is expiring in 2 days, so the brokers need a roll-restart to pick up the new one. As far as I can tell, this operation also requires topics to be balanced, which is not the case at the moment.
What's the estimation of the time it would take to rebalance kafka-main at least in eqiad as it stands?
Validating broker list:
Broker 1003 does not have a rack.id defined
Broker 1001 does not have a rack.id defined
Broker 1004 does not have a rack.id defined
Broker 1005 does not have a rack.id defined
Broker 1002 does not have a rack.id defined
-
Brokers targeted for partition offloading (< 1500.00GB storage free):
Reassignment parameters:
Ignoring partitions smaller than 512MB
Free storage mean, harmonic mean: 997.84GB, 939.58GB
Broker free storage limits (with a 8.00% tolerance from mean):
Sources limited to <= 1077.66GB
Destinations limited to >= 918.01GB
Broker 1001 relocations planned:
[51.10GB] eqiad.mediainfo-streaming-updater.mutation p0 -> 1005
[29.41GB] codfw.resource-purge p2 -> 1005
[21.07GB] codfw.mediawiki.job.wikibase-addUsagesForPage p0 -> 1005
[16.15GB] codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p5 -> 1005
[16.15GB] codfw.mediawiki.job.RecordLintJob p3 -> 1005
[16.05GB] eqiad.mediawiki.job.parsoidCachePrewarm p2 -> 1005
Broker 1003 relocations planned:
[31.02GB] codfw.resource-purge p1 -> 1005
[19.09GB] eqiad.rdf-streaming-updater.mutation-main p0 -> 1005
[17.18GB] codfw.mediawiki.job.RecordLintJob p0 -> 1005
[16.31GB] codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p0 -> 1005
[16.15GB] codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p4 -> 1005
[15.99GB] eqiad.mediawiki.job.parsoidCachePrewarm p1 -> 1005
Broker 1004 relocations planned:
[29.84GB] codfw.resource-purge p4 -> 1005
[23.28GB] eqiad.rdf-streaming-updater.mutation p0 -> 1005
[16.80GB] codfw.mediawiki.job.RecordLintJob p1 -> 1005
[16.27GB] codfw.mediawiki.job.RecordLintJob p4 -> 1005
[16.08GB] codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p7 -> 1005
[15.99GB] eqiad.mediawiki.job.parsoidCachePrewarm p0 -> 1005
Broker 1002 relocations planned:
[30.19GB] codfw.resource-purge p3 -> 1005
[10.55GB] codfw.mediawiki.page_change.v1 p0 -> 1005
-
Total relocation volume: 424.68GBcgoubert@kafka-main1006:~/T407185$ topicmappr rebalance --brokers -2 --topics '.*' --storage-threshold-gb 1500 --optimize-leadership --out-file rebalancing | grep -v no-op | grep -c 'preferred leader' 1206 cgoubert@kafka-main1006:~/T407185$ topicmappr rebalance --brokers -2 --topics '.*' --storage-threshold-gb 1500 --optimize-leadership --out-file rebalancing | grep -v no-op | grep -c 'replaced broker' 20
As stated to @Clement_Goubert on IRC: feel free to RR the cluster in its current state. Once the certificates are renewed, we can pair on the rebalancing.
I've started rebalancing kafka-logging-eqiad with the following overall plan:
Storage free change estimations: range: 728.00GB -> 32.82GB range spread: 56.07% -> 2.01% std. deviation: 281.23GB -> 11.26GB min-max: 1298.34GB, 2026.35GB -> 1636.37GB, 1669.19GB - Broker 1001: 1854.42 -> 1636.37 (-218.05GB, -11.76%) Broker 1002: 2026.35 -> 1653.47 (-372.88GB, -18.40%) Broker 1004: 1361.12 -> 1657.41 (+296.29GB, 21.77%) Broker 1005: 1720.27 -> 1669.19 (-51.08GB, -2.97%) Broker 1006: 1298.34 -> 1644.06 (+345.72GB, 26.63%)
Because kafka-logging has heterogeneous partition sizes, the leadership count is expected, as it leads to traffic and storage balance between brokers.
Waiting until T405950: eqiad row C/D Service Ops host migrations is done with moving the kafka-main nodes so we don't run into a network blip if the rebalance takes a while
Mentioned in SAL (#wikimedia-operations) [2025-11-26T12:06:42Z] <claime> Starting kafka-main rebalance with 30MB/s throttle - T407185
Rebalance done on kafka-main-eqiad
Will do kafka-main-codfw maybe tomorrow.
I've generated a rebalancing plan on kafka-main2008 using
brouberol@kafka-main2008:~/T407185$ topicmappr rebalance --topics '.*' --brokers -2 --out-file rebalancing --storage-threshold-gb 1000
Broker distribution: degree [min/max/avg]: 4/4/4.00 -> 4/4/4.00 - Broker 2001 - leader: 324, follower: 708, total: 1032 Broker 2002 - leader: 282, follower: 732, total: 1014 Broker 2003 - leader: 316, follower: 647, total: 963 Broker 2004 - leader: 146, follower: 263, total: 409 Broker 2005 - leader: 153, follower: 92, total: 245 Storage free change estimations: range: 504.35GB -> 15.70GB range spread: 59.85% -> 1.55% std. deviation: 187.10GB -> 5.77GB min-max: 842.68GB, 1347.03GB -> 1014.30GB, 1030.00GB - Broker 2001: 1111.64 -> 1020.61 (-91.03GB, -8.19%) Broker 2002: 868.96 -> 1030.00 (+161.04GB, 18.53%) Broker 2003: 842.68 -> 1022.32 (+179.64GB, 21.32%) Broker 2004: 945.92 -> 1028.99 (+83.08GB, 8.78%) Broker 2005: 1347.03 -> 1014.30 (-332.72GB, -24.70%)
@Clement_Goubert I'm happy to run the operation, or you can take it from here. You tell me!
Mentioned in SAL (#wikimedia-sre) [2025-12-02T13:04:22Z] <brouberol> running rebalancing of kafka-main-codfw with throttle of 30MB/s - T407185
brouberol@kafka-main2008:~/T407185$ kafka reassign-partitions kafka reassign-partitions --reassignment-json-file ./rebalancing.json --execute --throttle 30000000
kafka-reassign-partitions --zookeeper conf2004.codfw.wmnet,conf2005.codfw.wmnet,conf2006.codfw.wmnet/kafka/main-codfw kafka reassign-partitions --reassignment-json-file ./rebalancing.json --execute --throttle 30000000
Current partition replica assignment
{"version":1,"partitions":[{"topic":"codfw.mediawiki.job.RecordLintJob","partition":3,"replicas":[2005,2001,2002],"log_dirs":["any","any","any"]},{"topic":"codfw.rdf-streaming-updater.mutation","partition":0,"replicas":[2004,2003,2002],"log_dirs":["any","any","any"]},{"topic":"codfw.resource-purge","partition":4,"replicas":[2004,2003,2005],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.cirrussearch.page_rerender.v1","partition":1,"replicas":[2004,2001,2002],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.job.RecordLintJob","partition":1,"replicas":[2002,2005,2003],"log_dirs":["any","any","any"]},{"topic":"codfw.mediainfo-streaming-updater.mutation","partition":0,"replicas":[2002,2003,2004],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.job.parsoidCachePrewarm","partition":0,"replicas":[2003,2004,2001],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.job.RecordLintJob","partition":4,"replicas":[2004,2005,2003],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.page_change.v1","partition":0,"replicas":[2003,2004,2001],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.cirrussearch.page_rerender.v1","partition":4,"replicas":[2002,2003,2004],"log_dirs":["any","any","any"]},{"topic":"codfw.resource-purge","partition":1,"replicas":[2005,2001,2003],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.job.RecordLintJob","partition":0,"replicas":[2001,2004,2005],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.cirrussearch.page_rerender.v1","partition":2,"replicas":[2005,2001,2002],"log_dirs":["any","any","any"]},{"topic":"codfw.rdf-streaming-updater.mutation-main","partition":0,"replicas":[2002,2003,2004],"log_dirs":["any","any","any"]},{"topic":"codfw.resource-purge","partition":0,"replicas":[2003,2004,2005],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.cirrussearch.page_rerender.v1","partition":3,"replicas":[2001,2002,2003],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.job.parsoidCachePrewarm","partition":2,"replicas":[2005,2001,2002],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.cirrussearch.page_rerender.v1","partition":0,"replicas":[2003,2004,2001],"log_dirs":["any","any","any"]},{"topic":"codfw.resource-purge","partition":3,"replicas":[2002,2001,2004],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.job.RecordLintJob","partition":2,"replicas":[2004,2001,2002],"log_dirs":["any","any","any"]},{"topic":"codfw.mediawiki.job.parsoidCachePrewarm","partition":1,"replicas":[2004,2001,2002],"log_dirs":["any","any","any"]},{"topic":"codfw.resource-purge","partition":2,"replicas":[2004,2002,2003],"log_dirs":["any","any","any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 30000000 B/s
Successfully started reassignment of partitions.One thing to note: we're seeing a lot of traffic already coming out of kafka-main2010, and the rebalancing will add new partitions to the broker (not necessary leaders, but still). If the resulting traffic / broker is too skewed and that kafka-main2010 is sending out too much data compared to other brokers., we can set it as a replica instead of a leader of the largest topic in the cluster:
brouberol@kafka-main2008:~/T407185$ kafka topics --describe --topic codfw.mediawiki.job.htmlCacheUpdate kafka-topics --zookeeper conf2004.codfw.wmnet,conf2005.codfw.wmnet,conf2006.codfw.wmnet/kafka/main-codfw --describe --topic codfw.mediawiki.job.htmlCacheUpdate Topic:codfw.mediawiki.job.htmlCacheUpdate PartitionCount:1 ReplicationFactor:3 Configs: Topic: codfw.mediawiki.job.htmlCacheUpdate Partition: 0 Leader: 2005 Replicas: 2005,2003,2001 Isr: 2001,2003,2005
We can switch the replicas from 2005,2003,2001 to 2003,2005,2001 for example, to remove the consumer traffic away from that specific broker. Note: choose the least loaded broker in terms of MB/s out.
I've switched leadership for the codfw.mediawiki.job.htmlCacheUpdate topic.
brouberol@kafka-main2008:~/T407185$ topicmappr rebuild --topics codfw.mediawiki.page-links-change --brokers -1 --leader-evac-brokers 2005 --leader-evac-topics codfw.mediawiki.page-links-change --ignore-warns
Topics:
codfw.mediawiki.page-links-change
Broker change summary:
Broker 2005 does not have a rack.id defined
Broker 2002 does not have a rack.id defined
Broker 2003 does not have a rack.id defined
-
Replacing 0, added 0, missing 0, total count changed by 0
Action:
no-op
Partition map changes:
codfw.mediawiki.page-links-change p0: [2005 2002 2003] -> [2002 2005 2003] preferred leader
Broker distribution:
degree [min/max/avg]: 2/2/2.00 -> 2/2/2.00
-
Broker 2002 - leader: 1, follower: 0, total: 1
Broker 2003 - leader: 0, follower: 1, total: 1
Broker 2005 - leader: 0, follower: 1, total: 1
WARN:
3 provided broker(s) do(es) not have a rack.id defined
New partition maps:
codfw.mediawiki.page-links-change.json
brouberol@kafka-main2008:~/T407185$ kafka reassign-partitions --reassignment-json-file codfw.mediawiki.page-links-change.json --execute
kafka-reassign-partitions --zookeeper conf2004.codfw.wmnet,conf2005.codfw.wmnet,conf2006.codfw.wmnet/kafka/main-codfw --reassignment-json-file codfw.mediawiki.page-links-change.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"codfw.mediawiki.page-links-change","partition":0,"replicas":[2005,2002,2003],"log_dirs":["any","any","any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
brouberol@kafka-main2008:~/T407185$ kafka reassign-partitions --reassignment-json-file codfw.mediawiki.page-links-change.json --verify
kafka-reassign-partitions --zookeeper conf2004.codfw.wmnet,conf2005.codfw.wmnet,conf2006.codfw.wmnet/kafka/main-codfw --reassignment-json-file codfw.mediawiki.page-links-change.json --verify
Status of partition reassignment:
Reassignment of partition codfw.mediawiki.page-links-change-0 completed successfullyThat allowed the consumer traffic for the topic with the highest outbound traffic to be moved away from kafka-main2010, which was already the broker with the highest outbound traffic, to another broker.
