Page MenuHomePhabricator
Create Task
Maniphest T434562

Migrate stat hosts to Bookworm or later
Closed, ResolvedPublic
Actions

Description

Per this Slack conversation with @BTullis , we need to get these hosts on Bookworm or later before the end of September.

These hosts need a bit of special handling:

  • Reimages should take place in a maintenance window to minimize user impact.
  • We should use the reuse-parts partman recipes so we can preserve user data.

Creating this ticket to:

  • Update partman recipes
  • Schedule maintenance windows/contact stat host users
  • Reimage the hosts
  • Verify operation
  • Update docs/procedures as needed

Details

Other Assignee
bking
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
install_server: Generate the H750 stat host reuse recipe at install timeBtullisoperations/puppetproduction+69 -14
Add a new partman reuse recipe for hardware RAID two drive hostsBtullisoperations/puppetproduction+19 -1
Install amd_rocm version 6.1 on stat1010 after reimage to bookwormBtullisoperations/puppetproduction+1 -1
Install amd_rocm version 6.1 on stat1008 after reimage to bookwormBtullisoperations/puppetproduction+1 -1
install_server: Install grub on all RAID members in reuse recipesBtullisoperations/puppetproduction+12 -0
stat hosts: update partitioning to retain /srv dataBkingoperations/puppetproduction+16 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Tue, Aug 11, 2:51 PM
bking changed the task status from Open to In Progress.Tue, Aug 11, 9:14 PM
bking claimed this task.
bking triaged this task as Medium priority.
bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.
bking added a comment.EditedFri, Aug 14, 3:47 PM

Each stat host has a different disk configuration, I've posted the output of lsblk on each host here .

The next step is to write (or find/use) a partman recipe for each host that preserves the contents of /srv.

stat10[09-10].eqiad.wmnet: 2 drives, probably modules/install_server/files/autoinstall/partman/custom/reuse-analytics-raid1-2dev.cfg
stat1011.eqiad.wmnet: 4 drives, probably modules/install_server/files/autoinstall/partman/custom/reuse-analytics-stat-4dev.cfg.

I'll get a patch up shortly.

gerritbot added a comment.Fri, Aug 14, 4:01 PM

Change #1325921 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] stat hosts: update partitioning to retain /srv data

https://gerrit.wikimedia.org/r/1325921

gerritbot added a project: Patch-For-Review.Fri, Aug 14, 4:01 PM
nshahquinn-wmf subscribed.Fri, Aug 14, 8:35 PM
gerritbot added a comment.Mon, Aug 17, 2:39 PM

Change #1325921 merged by Bking:

[operations/puppet@production] stat hosts: update partitioning to retain /srv data

https://gerrit.wikimedia.org/r/1325921

Maintenance_bot removed a project: Patch-For-Review.Mon, Aug 17, 3:32 PM
BTullis updated the task description. (Show Details)Tue, Aug 18, 4:01 PM
BTullis added a comment.Tue, Aug 18, 5:02 PM

The proposed schedule is as follows:

stat1008 - Wednesday 19th @ 10:00 UTC
stat1009 - Wednesday 19th @ 16:00 UTC
stat1010 - Thursday 20th @ 10:00 UTC
stat1011 - Thursday 20th @ 16:00 UTC

BTullis claimed this task.Tue, Aug 18, 5:12 PM
BTullis updated the task description. (Show Details)
KCVelaga_WMF subscribed.Wed, Aug 19, 9:04 AM
ops-monitoring-bot added a comment.Wed, Aug 19, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host stat1008.eqiad.wmnet with OS bookworm

gerritbot added a comment.Wed, Aug 19, 11:04 AM

Change #1327088 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] install_server: Install grub on all RAID members in reuse recipes

https://gerrit.wikimedia.org/r/1327088

gerritbot added a project: Patch-For-Review.Wed, Aug 19, 11:04 AM
gerritbot added a comment.Wed, Aug 19, 11:06 AM

Change #1327088 merged by Btullis:

[operations/puppet@production] install_server: Install grub on all RAID members in reuse recipes

https://gerrit.wikimedia.org/r/1327088

ops-monitoring-bot added a comment.Wed, Aug 19, 11:07 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host stat1008.eqiad.wmnet with OS bookworm executed with errors:

  • stat1008 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console stat1008.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Wed, Aug 19, 11:18 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host stat1008.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Wed, Aug 19, 11:31 AM
BTullis added a comment.Wed, Aug 19, 12:15 PM

First puppet run on stat1008 has failed.

image.png (1,154×451 px, 63 KB)

Working on a puppet patch now.

gerritbot added a comment.Wed, Aug 19, 12:28 PM

Change #1327105 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Install amd_rocm version 6.1 on stat1008 after reimage to bookworm

https://gerrit.wikimedia.org/r/1327105

gerritbot added a project: Patch-For-Review.Wed, Aug 19, 12:28 PM
gerritbot added a comment.Wed, Aug 19, 12:31 PM

Change #1327105 merged by Btullis:

[operations/puppet@production] Install amd_rocm version 6.1 on stat1008 after reimage to bookworm

https://gerrit.wikimedia.org/r/1327105

BTullis added a subtask: T433348: Unresponsive management for an-worker1147.mgmt:22.Wed, Aug 19, 12:49 PM
BTullis removed a subtask: T433348: Unresponsive management for an-worker1147.mgmt:22.
BTullis added a parent task: T434494: Migrate production hadoop cluster to bookworm.
BTullis added a comment.Wed, Aug 19, 12:52 PM

I fixed the amd_rocm version number, but during the first puppet run I saw this on the console of stat1008.

Debian GNU/Linux 12 stat1008 ttyS1

stat1008 login: root
Password: 
[ 3589.672355] ---[ end Kernel panic - not syncing: GHES: Fatal hardware error ]---

Investigating.

Maintenance_bot removed a project: Patch-For-Review.Wed, Aug 19, 1:32 PM
MNeisler subscribed.Wed, Aug 19, 1:57 PM
MNeisler unsubscribed.
MNeisler subscribed.
ops-monitoring-bot added a comment.Wed, Aug 19, 1:57 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host stat1008.eqiad.wmnet with OS bookworm completed:

  • stat1008 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202608191148_btullis_1371291_stat1008.out, asking the operator what to do
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202608191207_btullis_1371291_stat1008.out, asking the operator what to do
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202608191233_btullis_1371291_stat1008.out, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608191306_btullis_1371291_stat1008.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BTullis mentioned this in T401692: EPIC: Migrate Data Platform SRE-owned hosts to Bookworm or later.Wed, Aug 19, 2:57 PM
BTullis mentioned this in T434494: Migrate production hadoop cluster to bookworm.Wed, Aug 19, 3:37 PM
ops-monitoring-bot added a comment.Thu, Aug 20, 10:23 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host stat1010.eqiad.wmnet with OS bookworm

gerritbot added a comment.Thu, Aug 20, 10:25 AM

Change #1327505 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Install amd_rocm version 6.1 on stat1010 after reimage to bookworm

https://gerrit.wikimedia.org/r/1327505

gerritbot added a project: Patch-For-Review.Thu, Aug 20, 10:25 AM

Change #1327505 merged by Btullis:

[operations/puppet@production] Install amd_rocm version 6.1 on stat1010 after reimage to bookworm

https://gerrit.wikimedia.org/r/1327505

Maintenance_bot removed a project: Patch-For-Review.Thu, Aug 20, 10:31 AM
gerritbot added a comment.Thu, Aug 20, 11:11 AM

Change #1327516 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add a new partman reuse recipe for hardware RAID two drive hosts

https://gerrit.wikimedia.org/r/1327516

gerritbot added a project: Patch-For-Review.Thu, Aug 20, 11:11 AM
gerritbot added a comment.Thu, Aug 20, 11:22 AM

Change #1327516 merged by Btullis:

[operations/puppet@production] Add a new partman reuse recipe for hardware RAID two drive hosts

https://gerrit.wikimedia.org/r/1327516

ops-monitoring-bot added a comment.Thu, Aug 20, 11:24 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host stat1010.eqiad.wmnet with OS bookworm executed with errors:

  • stat1010 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console stat1010.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Maintenance_bot removed a project: Patch-For-Review.Thu, Aug 20, 11:30 AM
ops-monitoring-bot added a comment.Thu, Aug 20, 11:33 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host stat1010.eqiad.wmnet with OS bookworm

gerritbot added a comment.Thu, Aug 20, 11:52 AM

Change #1327520 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] install_server: Generate the H750 stat host reuse recipe at install time

https://gerrit.wikimedia.org/r/1327520

gerritbot added a project: Patch-For-Review.Thu, Aug 20, 11:53 AM
gerritbot added a comment.Thu, Aug 20, 11:55 AM

Change #1327520 merged by Btullis:

[operations/puppet@production] install_server: Generate the H750 stat host reuse recipe at install time

https://gerrit.wikimedia.org/r/1327520

ops-monitoring-bot added a comment.Thu, Aug 20, 12:00 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host stat1010.eqiad.wmnet with OS bookworm executed with errors:

  • stat1010 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console stat1010.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, Aug 20, 12:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host stat1010.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, Aug 20, 12:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host stat1009.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Thu, Aug 20, 12:31 PM
ops-monitoring-bot added a comment.Thu, Aug 20, 1:21 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host stat1010.eqiad.wmnet with OS bookworm completed:

  • stat1010 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608201224_btullis_1894602_stat1010.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
OKarakaya-WMF subscribed.Thu, Aug 20, 1:26 PM
ops-monitoring-bot added a comment.Thu, Aug 20, 1:26 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host stat1009.eqiad.wmnet with OS bookworm completed:

  • stat1009 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608201232_btullis_1894737_stat1009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, Aug 20, 4:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2003 for host stat1011.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, Aug 20, 5:34 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2003 for host stat1011.eqiad.wmnet with OS bookworm completed:

  • stat1011 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608201643_bking_3391430_stat1011.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
bking closed this task as Resolved.Thu, Aug 20, 8:28 PM
bking updated the task description. (Show Details)
bking moved this task from In Progress to Done on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.

stat1011 successfully reimaged, which means all stat hosts have been successfully migrated to Bookworm. Closing...

bking updated Other Assignee, added: bking.Thu, Aug 20, 8:29 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.Fri, Aug 21, 7:58 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits