Page MenuHomePhabricator
Create Task
Maniphest T427088

[Post kafka-main 3.7 upgrade work] Reimage brokers to trixie/JDK21 & vlan migrations on select brokers
Closed, ResolvedPublic
Actions

Description

Splitting off the kafka-main Debian Trixie upgrade work from T419216: Upgrade kafka-main to Kafka 3.7 for better visibility
and consistency with T426835: Upgrade kafka-jumbo to trixie/JDK21 and T417001: Upgrade Observability Kafka-logging hosts to trixie

Having upgraded kafka-main to kafka 3.7, we're implementing the final part of the upgrade plan enacted by the Kafka upgrade WG, which is to upgrade the brokers to Debian Trixie / jdk21.

Additional migrations/upgrades:

codfw: T428191: ServiceOps: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets
eqiad: T421711: ServiceOps: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets

The VLAN migration practically means running the reimage cookbook with the --move-vlan flag. This will change the IP of the host which needs additional attention:

  • Update kafka_brokers_main in hieradata/common.yaml, run puppet on conf100* and the other brokers in eqiad
  • Update external-services in k8s.
    • Run puppet on deployment server(s)
    • Deploy the external service release to update network policies to include the new broker IP (and remove the old)
    • Roll restart eventgate-main out of precaution since it's pretty picky with loosing brokers

Post-upgrade cleanup:

  • Clean up per-host hiera configs, move to role level

Cluster states:

kafka-main codfw:

Kafka BrokerConfluent distribution 77Inter-broker protocolHost-level override patchDebian Trixie
kafka-main2006.codfw.wmnetUpgraded ✅3.7 ✅1288917✅
kafka-main2007.codfw.wmnetUpgraded ✅3.7 ✅1288918✅
kafka-main2008.codfw.wmnetUpgraded ✅3.7 ✅1288919✅
kafka-main2009.codfw.wmnetUpgraded ✅3.7 ✅1288920✅
kafka-main2010.codfw.wmnetUpgraded ✅3.7 ✅1288921✅

kafka-main eqiad:

Kafka BrokerConfluent distribution 77Inter-broker protocolHost-level override patchDebian Trixie
kafka-main1006.eqiad.wmnetUpgraded ✅3.7 ✅1285474✅
kafka-main1007.eqiad.wmnetUpgraded ✅3.7 ✅1285475✅
kafka-main1008.eqiad.wmnetUpgraded ✅3.7 ✅1285476✅
kafka-main1009.eqiad.wmnetUpgraded ✅3.7 ✅1285477✅
kafka-main1010.eqiad.wmnetUpgraded ✅3.7 ✅1285478✅

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
kafka-main: clean up host level overrides for kafka-main jdk 21 in codfwJasmineoperations/puppetproduction+0 -16
kafka-main: clean up host level overrides for kafka-main jdk 21 in eqiadJasmineoperations/puppetproduction+4 -20
hieradata/common.yaml: add new IPs for kafka-main1009 following vlan migrationJasmineoperations/puppetproduction+2 -2
kafka-main2010: apply host-level override in advance of trixie upgrade [0]Jasmineoperations/puppetproduction+4 -0
hieradata/common.yaml: add new IPs for kafka-main2010, following vlan migrationJasmineoperations/puppetproduction+2 -2
hieradata/common.yaml: add new IPs for kafka-main1008, following vlan migrationsJasmineoperations/puppetproduction+2 -2
hieradata/common.yaml: add new IPs for kafka-main2009, following vlan migrationsJasmineoperations/puppetproduction+2 -2
kafka-main2009: apply host-level override in advance of trixie upgrade [0]Jasmineoperations/puppetproduction+4 -0
hieradata/common.yaml: add new IPs for kafka-main2008, following vlan migrationsJasmineoperations/puppetproduction+2 -2
kafka-main2008: apply host-level override in advance of trixie upgrade [0]Jasmineoperations/puppetproduction+4 -0
kafka-main2007: apply host-level override in advance of trixie upgrade [0]Jasmineoperations/puppetproduction+4 -0
reuse-raid10-6dev.cfg: Fix swap reuse and grub-install on all disksJMeybohmoperations/puppetproduction+8 -5
partman/reuse-raid10-6dev.cfg: Apply swap workaroundJMeybohmoperations/puppetproduction+2 -1
partman/reuse-raid10-6dev.cfg: Apply workaround to swap handling affecting trixie installationsJMeybohmoperations/puppetproduction+3 -0
partman/reuse-raid10-6dev.cfg: Use linux-swap as fs identifierJMeybohmoperations/puppetproduction+1 -1
kafka-main2006: apply host-level override in advance of trixie upgrade [0]Jasmineoperations/puppetproduction+4 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
jasmine_ renamed this task from Upgrade kafka-main brokers to Trixie/JDK21 to [Post kafka 3.7 upgrade work] Reimage brokers to trixie/JDK21 & vlan migrations on select brokers.May 22 2026, 5:48 PM
jasmine_ renamed this task from [Post kafka 3.7 upgrade work] Reimage brokers to trixie/JDK21 & vlan migrations on select brokers to [Post kafka 3.7 on kafka-main upgrade work] Reimage brokers to trixie/JDK21 & vlan migrations on select brokers.
jasmine_ updated the task description. (Show Details)
jasmine_ updated the task description. (Show Details)May 22 2026, 5:50 PM
jasmine_ renamed this task from [Post kafka 3.7 on kafka-main upgrade work] Reimage brokers to trixie/JDK21 & vlan migrations on select brokers to [Post kafka-main 3.7 upgrade work] Reimage brokers to trixie/JDK21 & vlan migrations on select brokers.May 22 2026, 5:58 PM
jasmine_ updated the task description. (Show Details)
gerritbot added a comment.May 22 2026, 6:05 PM

Change #1288917 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] kafka-main2006: apply host-level override in advance of trixie upgrade [0]

https://gerrit.wikimedia.org/r/1288917

gerritbot added a project: Patch-For-Review.May 22 2026, 6:05 PM

Change #1288918 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] kafka-main2007: apply host-level override in advance of trixie upgrade [0]

https://gerrit.wikimedia.org/r/1288918

gerritbot added a comment.May 22 2026, 6:08 PM

Change #1288919 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] kafka-main2008: apply host-level override in advance of trixie upgrade [0]

https://gerrit.wikimedia.org/r/1288919

gerritbot added a comment.May 22 2026, 6:08 PM

Change #1288920 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] kafka-main2009: apply host-level override in advance of trixie upgrade [0]

https://gerrit.wikimedia.org/r/1288920

gerritbot added a comment.May 22 2026, 6:09 PM

Change #1288921 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] kafka-main2010: apply host-level override in advance of trixie upgrade [0]

https://gerrit.wikimedia.org/r/1288921

jijiki moved this task from Inbox to In Progress on the ServiceOps new board.May 26 2026, 8:11 AM
JMeybohm subscribed.May 26 2026, 9:20 AM
gerritbot added a comment.Jun 1 2026, 4:52 PM

Change #1288917 merged by Jasmine:

[operations/puppet@production] kafka-main2006: apply host-level override in advance of trixie upgrade [0]

https://gerrit.wikimedia.org/r/1288917

ops-monitoring-bot added a comment.Jun 1 2026, 4:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Jun 1 2026, 5:53 PM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie executed with errors:

  • kafka-main2006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console kafka-main2006.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
JMeybohm added a comment.Jun 2 2026, 10:52 AM
In T427088#11973521, @ops-monitoring-bot wrote:

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie executed with errors:

The debian installer is hanging in partman, asking if we're sure to set the host up without swap. Maybe there is a problem with the partman receipt since the md has been assembled correctly and the logical volumes have been scanned as well:

~ # cat /proc/mdstat 
Personalities : [raid10] 
md0 : active raid10 sdb2[0] sdf2[5] sde2[4] sdd2[3] sdc2[2] sda2[1]
      2811801600 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]
      bitmap: 0/21 pages [0KB], 65536KB chunk

unused devices: <none>
~ # lvdisplay 
  --- Logical volume ---
  LV Path                /dev/vg0/swap
  LV Name                swap
  VG Name                vg0
  LV UUID                2TWWua-46AX-ckM8-bokB-uer3-mSWu-e84XGo
  LV Write Access        read/write
  LV Creation host, time kafka-main2006, 2024-08-15 08:12:46 +0000
  LV Status              available
  # open                 0
  LV Size                976.00 MiB
  Current LE             244
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8386560
  Block device           253:0
   
  --- Logical volume ---
  LV Path                /dev/vg0/root
  LV Name                root
  VG Name                vg0
  LV UUID                kB47RB-iMRc-zwxl-7O0X-qONV-OtkS-m9hQAb
  LV Write Access        read/write
  LV Creation host, time kafka-main2006, 2024-08-15 08:12:47 +0000
  LV Status              available
  # open                 0
  LV Size                74.50 GiB
  Current LE             19073
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8386560
  Block device           253:1
   
  --- Logical volume ---
  LV Path                /dev/vg0/srv
  LV Name                srv
  VG Name                vg0
  LV UUID                6SIWCc-Ay9g-f0DU-Pn8S-lhga-xCTQ-thBZUn
  LV Write Access        read/write
  LV Creation host, time kafka-main2006, 2024-08-15 08:12:47 +0000
  LV Status              available
  # open                 0
  LV Size                2.02 TiB
  Current LE             529861
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8386560
  Block device           253:2
gerritbot added a comment.Jun 2 2026, 11:23 AM

Change #1296553 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] partman/reuse-raid10-6dev.cfg: Use linux-swap as fs identifier

https://gerrit.wikimedia.org/r/1296553

JMeybohm added a comment.Jun 2 2026, 12:07 PM

The patch might be not enough (will test anyways). If not, we might need to add the workaround from T408777: Swap partition issue when installing a DB with Debian Trixie

gerritbot added a comment.Jun 2 2026, 12:08 PM

Change #1296553 merged by JMeybohm:

[operations/puppet@production] partman/reuse-raid10-6dev.cfg: Use linux-swap as fs identifier

https://gerrit.wikimedia.org/r/1296553

ops-monitoring-bot added a comment.Jun 2 2026, 12:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Jun 2 2026, 1:04 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie executed with errors:

  • kafka-main2006 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console kafka-main2006.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jun 2 2026, 1:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie

gerritbot added a comment.Jun 2 2026, 1:45 PM

Change #1296597 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] partman/reuse-raid10-6dev.cfg: Apply workaround to swap handling affecting trixie installations

https://gerrit.wikimedia.org/r/1296597

gerritbot added a comment.Jun 2 2026, 2:10 PM

Change #1296597 merged by JMeybohm:

[operations/puppet@production] partman/reuse-raid10-6dev.cfg: Apply workaround to swap handling affecting trixie installations

https://gerrit.wikimedia.org/r/1296597

ops-monitoring-bot added a comment.Jun 2 2026, 2:15 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie executed with errors:

  • kafka-main2006 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console kafka-main2006.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jun 2 2026, 2:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie

gerritbot added a comment.Jun 2 2026, 3:11 PM

Change #1296613 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] partman/reuse-raid10-6dev.cfg: Apply swap workaround

https://gerrit.wikimedia.org/r/1296613

gerritbot added a comment.Jun 2 2026, 3:12 PM

Change #1296613 merged by JMeybohm:

[operations/puppet@production] partman/reuse-raid10-6dev.cfg: Apply swap workaround

https://gerrit.wikimedia.org/r/1296613

ops-monitoring-bot added a comment.Jun 2 2026, 3:12 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie executed with errors:

  • kafka-main2006 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console kafka-main2006.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jun 2 2026, 3:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie

JMeybohm added a comment.Jun 2 2026, 4:21 PM

With the two workarounds the installer does on longer hang on asking for swap. Unfortunately it still failed on first try to create a filesystem for the root partition (without saying so ofc). Restarting the reimage did work but it did not include the swap partition in /etc/fstab (and did not activate it).
I will complete the reimage so we have the broker back and it can catch up with what it missed (since /srv seems to not have been formatted, yay) but before taking on the next broker we should figure out what's happening/not happening here.

Maybe it is possible to test the recipe with a 6 disk ganeti vm to have a workable feedback loop time and not interfere with the kafka-main cluster.

ops-monitoring-bot added a comment.Jun 2 2026, 4:27 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie completed:

  • kafka-main2006 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606021609_jayme_3605330_kafka-main2006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
JMeybohm mentioned this in T428078: Swap partition not used when reimaging to trixie with reuse-parts.cfg and mdadm + lvm.Jun 3 2026, 4:06 PM
gerritbot added a comment.Jun 3 2026, 5:16 PM

Change #1297201 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] reuse-raid10-6dev.cfg: Fix swap reuse and grub-install on all disks

https://gerrit.wikimedia.org/r/1297201

MLechvien-WMF added a subtask: T428078: Swap partition not used when reimaging to trixie with reuse-parts.cfg and mdadm + lvm.Jun 4 2026, 8:46 AM
gerritbot added a comment.Jun 4 2026, 9:52 AM

Change #1297201 merged by JMeybohm:

[operations/puppet@production] reuse-raid10-6dev.cfg: Fix swap reuse and grub-install on all disks

https://gerrit.wikimedia.org/r/1297201

gerritbot added a comment.Jun 4 2026, 2:53 PM

Change #1288918 merged by JMeybohm:

[operations/puppet@production] kafka-main2007: apply host-level override in advance of trixie upgrade [0]

https://gerrit.wikimedia.org/r/1288918

ops-monitoring-bot added a comment.Jun 4 2026, 2:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host kafka-main2007.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Jun 4 2026, 3:41 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host kafka-main2007.codfw.wmnet with OS trixie completed:

  • kafka-main2007 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606041524_jayme_1135974_kafka-main2007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
JMeybohm closed subtask T428078: Swap partition not used when reimaging to trixie with reuse-parts.cfg and mdadm + lvm as Resolved.Jun 4 2026, 3:47 PM
JMeybohm updated the task description. (Show Details)
JMeybohm added a comment.Jun 4 2026, 3:49 PM

The fix from T428078: Swap partition not used when reimaging to trixie with reuse-parts.cfg and mdadm + lvm works fine so we may continue with the reimages as planned @jasmine_

gerritbot added a comment.Jun 4 2026, 4:06 PM

Change #1288919 merged by Jasmine:

[operations/puppet@production] kafka-main2008: apply host-level override in advance of trixie upgrade [0]

https://gerrit.wikimedia.org/r/1288919

jasmine_ added a comment.EditedJun 4 2026, 4:13 PM
In T427088#11986067, @JMeybohm wrote:

The fix from T428078: Swap partition not used when reimaging to trixie with reuse-parts.cfg and mdadm + lvm works fine so we may continue with the reimages as planned @jasmine_

Thanks Janis! Much appreciated~ Proceeding with remaining brokers momentarily.

Edit: It appears that all remaining brokers in codfw kafka-main are on legacy vlan, captured in T428191: ServiceOps: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets.
We will need to move vlan on these as with kafka-main100[8-9].

For the time being, will proceed with brokers on the non legacy vlan in eqiad, then to the brokers requiring vlan migration following.

ops-monitoring-bot added a comment.Jun 4 2026, 11:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main1006.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jun 4 2026, 11:57 PM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main1006.eqiad.wmnet with OS trixie completed:

  • kafka-main1006 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606042340_jasmine_86465_kafka-main1006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jun 5 2026, 12:17 AM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main1007.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jun 5 2026, 12:56 AM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main1007.eqiad.wmnet with OS trixie completed:

  • kafka-main1007 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606050040_jasmine_100324_kafka-main1007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jun 5 2026, 1:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main1010.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jun 5 2026, 1:55 AM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main1010.eqiad.wmnet with OS trixie completed:

  • kafka-main1010 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606050139_jasmine_114239_kafka-main1010.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jasmine_ updated the task description. (Show Details)Jun 5 2026, 2:03 AM
jasmine_ added a comment.EditedJun 5 2026, 3:11 AM
In T427088#11986143, @jasmine_ wrote:
In T427088#11986067, @JMeybohm wrote:

The fix from T428078: Swap partition not used when reimaging to trixie with reuse-parts.cfg and mdadm + lvm works fine so we may continue with the reimages as planned @jasmine_

Thanks Janis! Much appreciated~ Proceeding with remaining brokers momentarily.

Edit: It appears that all remaining brokers in codfw kafka-main are on legacy vlan, captured in T428191: ServiceOps: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets.
We will need to move vlan on these as with kafka-main100[8-9].

For the time being, will proceed with brokers on the non legacy vlan in eqiad, then to the brokers requiring vlan migration following.

All kafka-main brokers on non-legacy vlan are now on trixie in both DCs. The remaining brokers will require a vlan migration prior to the trixie reimage.

kafka-main100[8-9], kafka-main200[8-9], kafka-main2010

codfw: T428191: ServiceOps: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets
eqiad: T421711: ServiceOps: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets

Since the vlan migrations will require a deployment, I plan to pick them up starting Monday.
Note: Additionally the cleanup for host level overrides for the eqiad cluster will be applied in kafka-main1009 as it will be the final broker to be reimaged to trixie (rather than kafka-main1010)

JMeybohm mentioned this in T428191: ServiceOps: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets.Jun 5 2026, 8:28 AM
MLechvien-WMF mentioned this in T419212: Upgrade ServiceOps roles from Bullseye to Debian Trixie.Jun 9 2026, 9:27 AM
ops-monitoring-bot added a comment.Jun 9 2026, 4:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main2008.codfw.wmnet with OS trixie

gerritbot added a comment.Jun 9 2026, 4:42 PM

Change #1299570 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] hieradata/common.yaml: add new IPs for kafka-main2008, following vlan migrations

https://gerrit.wikimedia.org/r/1299570

gerritbot added a comment.Jun 9 2026, 4:56 PM

Change #1299570 merged by Jasmine:

[operations/puppet@production] hieradata/common.yaml: add new IPs for kafka-main2008, following vlan migrations

https://gerrit.wikimedia.org/r/1299570

JMeybohm created subtask T428654: kafka-main2008 CPU 2 MEMABCD VPP PG voltage is outside of range..Jun 9 2026, 5:34 PM
ops-monitoring-bot added a comment.Jun 9 2026, 6:26 PM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main2008.codfw.wmnet with OS trixie executed with errors:

  • kafka-main2008 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console kafka-main2008.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jun 9 2026, 6:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main2008.codfw.wmnet with OS trixie

JMeybohm closed subtask T428654: kafka-main2008 CPU 2 MEMABCD VPP PG voltage is outside of range. as Resolved.Jun 9 2026, 7:13 PM
ops-monitoring-bot added a comment.Jun 9 2026, 7:15 PM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main2008.codfw.wmnet with OS trixie completed:

  • kafka-main2008 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606091858_jasmine_1492495_kafka-main2008.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jun 10 2026, 12:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main2009.codfw.wmnet with OS trixie

gerritbot added a comment.Jun 10 2026, 12:04 AM

Change #1288920 merged by Jasmine:

[operations/puppet@production] kafka-main2009: apply host-level override in advance of trixie upgrade [0]

https://gerrit.wikimedia.org/r/1288920

gerritbot added a comment.Jun 10 2026, 12:07 AM

Change #1299656 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] hieradata/common.yaml: add new IPs for kafka-main2009, following vlan migrations

https://gerrit.wikimedia.org/r/1299656

gerritbot added a comment.Jun 10 2026, 12:09 AM

Change #1299656 merged by Jasmine:

[operations/puppet@production] hieradata/common.yaml: add new IPs for kafka-main2009, following vlan migrations

https://gerrit.wikimedia.org/r/1299656

ops-monitoring-bot added a comment.Jun 10 2026, 1:00 AM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main2009.codfw.wmnet with OS trixie completed:

  • kafka-main2009 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606100043_jasmine_1563623_kafka-main2009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jasmine_ updated the task description. (Show Details)Jun 10 2026, 1:01 AM
ops-monitoring-bot added a comment.Jun 10 2026, 1:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main1008.eqiad.wmnet with OS trixie

gerritbot added a comment.Jun 10 2026, 1:15 AM

Change #1299661 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] hieradata/common.yaml: add new IPs for kafka-main1008, following vlan migrations

https://gerrit.wikimedia.org/r/1299661

gerritbot added a comment.Jun 10 2026, 1:16 AM

Change #1299661 merged by Jasmine:

[operations/puppet@production] hieradata/common.yaml: add new IPs for kafka-main1008, following vlan migrations

https://gerrit.wikimedia.org/r/1299661

ops-monitoring-bot added a comment.Jun 10 2026, 2:07 AM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main1008.eqiad.wmnet with OS trixie completed:

  • kafka-main1008 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606100149_jasmine_1578727_kafka-main1008.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jasmine_ updated the task description. (Show Details)Jun 10 2026, 2:11 AM
ops-monitoring-bot added a comment.Jun 10 2026, 4:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main2010.codfw.wmnet with OS trixie

gerritbot added a comment.Jun 10 2026, 4:54 PM

Change #1300208 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] hieradata/common.yaml: add new IPs for kafka-main2010, following vlan migration

https://gerrit.wikimedia.org/r/1300208

gerritbot added a comment.Jun 10 2026, 4:56 PM

Change #1300208 merged by Jasmine:

[operations/puppet@production] hieradata/common.yaml: add new IPs for kafka-main2010, following vlan migration

https://gerrit.wikimedia.org/r/1300208

gerritbot added a comment.Jun 10 2026, 4:56 PM

Change #1288921 merged by Jasmine:

[operations/puppet@production] kafka-main2010: apply host-level override in advance of trixie upgrade [0]

https://gerrit.wikimedia.org/r/1288921

Maintenance_bot removed a project: Patch-For-Review.Jun 10 2026, 5:31 PM
ops-monitoring-bot added a comment.Jun 10 2026, 5:46 PM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main2010.codfw.wmnet with OS trixie completed:

  • kafka-main2010 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606101729_jasmine_1790524_kafka-main2010.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jasmine_ updated the task description. (Show Details)Jun 10 2026, 5:47 PM
ops-monitoring-bot added a comment.Jun 11 2026, 12:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main1009.eqiad.wmnet with OS trixie

gerritbot added a comment.Jun 11 2026, 12:34 AM

Change #1300281 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] hieradata/common.yaml: add new IPs for kafka-main1009 following vlan migration

https://gerrit.wikimedia.org/r/1300281

gerritbot added a project: Patch-For-Review.Jun 11 2026, 12:34 AM

Change #1300281 merged by Jasmine:

[operations/puppet@production] hieradata/common.yaml: add new IPs for kafka-main1009 following vlan migration

https://gerrit.wikimedia.org/r/1300281

ops-monitoring-bot added a comment.Jun 11 2026, 1:18 AM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main1009.eqiad.wmnet with OS trixie completed:

  • kafka-main1009 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606110102_jasmine_1891761_kafka-main1009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jasmine_ updated the task description. (Show Details)Jun 11 2026, 1:20 AM
gerritbot added a comment.Jun 11 2026, 1:30 AM

Change #1300287 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] kafka-main: clean up host level overrides for kafka-main jdk 21 in eqiad

https://gerrit.wikimedia.org/r/1300287

gerritbot added a comment.Jun 11 2026, 1:35 AM

Change #1300288 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/puppet@production] kafka-main: clean up host level overrides for kafka-main jdk 21 in codfw

https://gerrit.wikimedia.org/r/1300288

jasmine_ updated the task description. (Show Details)Jun 11 2026, 2:07 AM
gerritbot added a comment.Jun 11 2026, 9:49 PM

Change #1300287 merged by Jasmine:

[operations/puppet@production] kafka-main: clean up host level overrides for kafka-main jdk 21 in eqiad

https://gerrit.wikimedia.org/r/1300287

gerritbot added a comment.Jun 11 2026, 9:49 PM

Change #1300288 merged by Jasmine:

[operations/puppet@production] kafka-main: clean up host level overrides for kafka-main jdk 21 in codfw

https://gerrit.wikimedia.org/r/1300288

jasmine_ updated the task description. (Show Details)Jun 11 2026, 9:53 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 11 2026, 10:31 PM
jasmine_ closed this task as Resolved.Jun 13 2026, 12:18 AM

Resolving as all kafka-main brokers are now on Trixie and legacy vlan brokers have also been migrated :) Thanks @JMeybohm for supporting!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits