Now the kafka clusters have all been upgraded to kafka 3.7, we're implementing the last part of the upgrade plan enacted by the Kafka upgrade WG, which is to upgrade the brokers to Debian Trixie / jdk21.
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • elukey | T416669 Upgrade Kafka to version 3.x | |||
| Resolved | brouberol | T426835 Upgrade kafka-jumbo to trixie/JDK21 |
Event Timeline
Comment Actions
Change #1289908 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] preseed: drop configuration for kafka-jumbo100[7-9]
Comment Actions
Change #1289909 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] preseed: prepare kafka-jumbo brokers for re-imaging
Comment Actions
Change #1289910 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] kafka-jumbo: upgrade broker to jdk21
Comment Actions
Change #1289908 merged by Brouberol:
[operations/puppet@production] preseed: drop configuration for kafka-jumbo100[7-9]
Comment Actions
Change #1289909 merged by Brouberol:
[operations/puppet@production] preseed: prepare kafka-jumbo brokers for re-imaging
Comment Actions
Change #1289910 merged by Brouberol:
[operations/puppet@production] kafka-jumbo: upgrade kafka-jumbo1010 to jdk21
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1010.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1010.eqiad.wmnet with OS trixie completed:
- kafka-jumbo1010 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605201119_brouberol_1969993_kafka-jumbo1010.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1289933 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade kafka-jumbo1011 to JDK21
Comment Actions
Change #1289933 merged by Brouberol:
[operations/puppet@production] Upgrade kafka-jumbo1011 to JDK21
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1011.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1011.eqiad.wmnet with OS trixie completed:
- kafka-jumbo1011 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605201225_brouberol_2020864_kafka-jumbo1011.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1289945 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade kafka-jumbo1012 to JDK21
Comment Actions
Change #1289946 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade kafka-jumbo1013 to JDK21
Comment Actions
Change #1289947 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade kafka-jumbo1014 to JDK21
Comment Actions
Change #1289948 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade kafka-jumbo1015 to JDK21
Comment Actions
Change #1289949 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade kafka-jumbo1016 to JDK21
Comment Actions
Change #1289951 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade kafka-jumbo1018 to JDK21
Comment Actions
Change #1289950 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade kafka-jumbo1017 to JDK21
Comment Actions
Change #1289945 merged by Brouberol:
[operations/puppet@production] Upgrade kafka-jumbo1012 to JDK21
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1012.eqiad.wmnet with OS trixie
Comment Actions
Change #1289959 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Set JDK21 as default for all kafka-jumbo brokers
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1012.eqiad.wmnet with OS trixie completed:
- kafka-jumbo1012 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605201323_brouberol_2041418_kafka-jumbo1012.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1289946 merged by Brouberol:
[operations/puppet@production] Upgrade kafka-jumbo1013 to JDK21
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1013.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1013.eqiad.wmnet with OS trixie completed:
- kafka-jumbo1013 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605201409_brouberol_2070890_kafka-jumbo1013.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1289947 merged by Brouberol:
[operations/puppet@production] Upgrade kafka-jumbo1014 to JDK21
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1014.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1014.eqiad.wmnet with OS trixie completed:
- kafka-jumbo1014 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605201504_brouberol_2117110_kafka-jumbo1014.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1289948 merged by Brouberol:
[operations/puppet@production] Upgrade kafka-jumbo1015 to JDK21
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1015.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1015.eqiad.wmnet with OS trixie completed:
- kafka-jumbo1015 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605201549_brouberol_2157224_kafka-jumbo1015.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1289949 merged by Brouberol:
[operations/puppet@production] Upgrade kafka-jumbo1016 to JDK21
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1016.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1016.eqiad.wmnet with OS trixie executed with errors:
- kafka-jumbo1016 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console kafka-jumbo1016.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1016.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1016.eqiad.wmnet with OS trixie executed with errors:
- kafka-jumbo1016 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console kafka-jumbo1016.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1016.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1016.eqiad.wmnet with OS trixie executed with errors:
- kafka-jumbo1016 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console kafka-jumbo1016.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Comment Actions
Change #1289950 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade kafka-jumbo1017 to JDK21
Comment Actions
Change #1289950 merged by Brouberol:
[operations/puppet@production] Upgrade kafka-jumbo1017 to JDK21
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1017.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1017.eqiad.wmnet with OS trixie completed:
- kafka-jumbo1017 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605211219_brouberol_2632541_kafka-jumbo1017.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1289951 merged by Brouberol:
[operations/puppet@production] Upgrade kafka-jumbo1018 to JDK21
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host kafka-jumbo1018.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host kafka-jumbo1018.eqiad.wmnet with OS trixie completed:
- kafka-jumbo1018 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605211259_brouberol_2681218_kafka-jumbo1018.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1289959 merged by Brouberol:
[operations/puppet@production] Set JDK21 as default for all kafka-jumbo brokers