We need to configure these hosts as kafka brokers, as well as reassign the data from kafka-jumbo100[7-9] to them.
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Unknown Object (Task) | |||||
| Resolved | Jclark-ctr | T377874 Q2:rack/setup/install kafka-jumbo10[16-18] | |||
| Resolved | None | T397447 Take kafka-jumbo100[7-9] out of service, ready for decom | |||
| Resolved | brouberol | T398826 Bring kafka-jumbo101[6-8] into service |
Event Timeline
Comment Actions
We can see that the disks were correctly configured, and that the large 18TB partition is available on all 3 hosts.
brouberol@cumin2002:~$ sudo cumin 'kafka-jumbo101[6-8].eqiad.wmnet' 'df -h' 3 hosts will be targeted: kafka-jumbo[1016-1018].eqiad.wmnet OK to proceed on 3 hosts? Enter the number of affected hosts to confirm or "q" to quit: 3 ===== NODE GROUP ===== (3) kafka-jumbo[1016-1018].eqiad.wmnet ----- OUTPUT of 'df -h' ----- Filesystem Size Used Avail Use% Mounted on udev 126G 0 126G 0% /dev tmpfs 26G 2.0M 26G 1% /run /dev/mapper/vg0-root 351G 5.9G 327G 2% / tmpfs 126G 0 126G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/mapper/vg1-srv 18T 24K 17T 1% /srv tmpfs 26G 0 26G 0% /run/user/0
Comment Actions
Change #1166832 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Provision kafka-jumbo1016
Comment Actions
Change #1166833 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Provision kafka-jumbo1017
Comment Actions
Change #1166834 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Provision kafka-jumbo1018
Comment Actions
Change #1166832 merged by Brouberol:
[operations/puppet@production] Provision kafka-jumbo1016
Comment Actions
Mentioned in SAL (#wikimedia-analytics) [2025-07-07T14:19:04Z] <brouberol> running a rolling-restart of kafka-jumbo - T398826
Comment Actions
Change #1166833 merged by Brouberol:
[operations/puppet@production] Provision kafka-jumbo1017
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2025-07-30T12:19:14Z] <brouberol> kafka-jumbo1017 is added to the cluster, puppet ran on all kafka/zookeeper hosts, external-services was updated on dse-k8s-eqiad, codfw and eqiad - T398826
Comment Actions
Mentioned in SAL (#wikimedia-analytics) [2025-07-30T12:19:20Z] <brouberol> kafka-jumbo1017 is added to the cluster, puppet ran on all kafka/zookeeper hosts, external-services was updated on dse-k8s-eqiad, codfw and eqiad - T398826
Comment Actions
Change #1166834 merged by Brouberol:
[operations/puppet@production] Provision kafka-jumbo1018
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2025-07-30T13:53:44Z] <brouberol> kafka-jumbo1018 is added to the cluster, puppet ran on all kafka/zookeeper hosts, external-services was updated on dse-k8s-eqiad, codfw and eqiad - T398826