Complete upgrading WMCS bare metal hosts to Trixie
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fnegri
	Sep 19 2024, 4:08 PM

Project Tags

Referenced Files

None

Subscribers

Description

We still have many WMCS hosts running Debian Bullseye or Bookworm. We should slowly upgrade all of them to Trixie. Bullseye reaches end of life on 31st Aug 2026

Please create sub-tasks for the various host groups.

root@cumin1002:~# cumin cloud* 'grep -i version= /etc/os-release'
151 hosts will be targeted:
cloudbackup[1001-1002]-dev.eqiad.wmnet,cloudbackup[2003-2004].codfw.wmnet,cloudbackup[1003-1004].eqiad.wmnet,cloudcephm
on[2004-2006]-dev.codfw.wmnet,cloudcephmon[1004-1006].eqiad.wmnet,cloudcephosd[2004-2007]-dev.codfw.wmnet,cloudcephosd[1016-1024,1026-1052].eqiad.wmnet,cloudcontrol[2005-2006,2010]-dev.codfw.wmnet,cloudcontrol[1006-1007,1011].eqiad.wmnet,cloudcumin2001.codfw.wmnet,cloudcumin1001.eqiad.wmnet,clouddb[1013-1020,1022-1025].eqiad.wmnet,clouddumps[1001-1002].wikimedia.org,cloudelastic[1007-1012].eqiad.wmnet,cloudgw[2002-2003]-dev.codfw.wmnet,cloudgw[1003-1004].eqiad.wmnet,cloudlb[2002-2004]-dev.codfw.wmnet,cloudlb[1001-1002].eqiad.wmnet,cloudnet[2005-2008]-dev.codfw.wmnet,cloudnet[1005-1006].eqiad.wmnet,cloudrabbit[2001-2003]-dev.codfw.wmnet,cloudrabbit[1001-1003].eqiad.wmnet,cloudservices[2004-2005]-dev.codfw.wmnet,cloudservices[1005-1006].eqiad.wmnet,cloudvirt[2004-2006]-dev.codfw.wmnet,cloudvirt[1040-1076].eqiad.wmnet,cloudvirtlocal[1001-1003].eqiad.wmnet,cloudweb2002-dev.wikimedia.org,cloudweb[1003-1004].wikimedia.org                       OK to proceed on 151 hosts? Enter the number of affected hosts to confirm or "q" to quit: 151
===== NODE GROUP =====                                                                                                 
(61) cloudcephmon[2004-2006]-dev.codfw.wmnet,cloudcephmon[1005-1006].eqiad.wmnet,cloudcephosd[2005-2007]-dev.codfw.wmnet,cloudcephosd[1016-1024,1026-1052].eqiad.wmnet,cloudcumin2001.codfw.wmnet,cloudcumin1001.eqiad.wmnet,clouddumps[1001-1
002].wikimedia.org,cloudelastic[1007-1012].eqiad.wmnet,cloudgw[2002-2003]-dev.codfw.wmnet,cloudgw[1003-1004].eqiad.wmnet,cloudweb2002-dev.wikimedia.org,cloudweb[1003-1004].wikimedia.org                                                     ----- OUTPUT of 'grep -i version= /etc/os-release' -----                                                               
VERSION="11 (bullseye)"                                                                                                
===== NODE GROUP =====                                                                                                 
(90) cloudbackup[1001-1002]-dev.eqiad.wmnet,cloudbackup[2003-2004].codfw.wmnet,cloudbackup[1003-1004].eqiad.wmnet,cloudcephmon1004.eqiad.wmnet,cloudcephosd2004-dev.codfw.wmnet,cloudcontrol[2005-2006,2010]-dev.codfw.wmnet,cloudcontrol[1006
-1007,1011].eqiad.wmnet,clouddb[1013-1020,1022-1025].eqiad.wmnet,cloudlb[2002-2004]-dev.codfw.wmnet,cloudlb[1001-1002].eqiad.wmnet,cloudnet[2005-2008]-dev.codfw.wmnet,cloudnet[1005-1006].eqiad.wmnet,cloudrabbit[2001-2003]-dev.codfw.wmnet,cloudrabbit[1001-1003].eqiad.wmnet,cloudservices[2004-2005]-dev.codfw.wmnet,cloudservices[1005-1006].eqiad.wmnet,cloudvirt[2004-2006]-dev.codfw.wmnet,cloudvirt[1040-1076].eqiad.wmnet,cloudvirtlocal[1001-1003].eqiad.wmnet                  ----- OUTPUT of 'grep -i version= /etc/os-release' -----                                                               
VERSION="12 (bookworm)"                                                                                                
================

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
cloudbackup: flip all backups from cloudbackup1004 to 1003	operations/puppet	production	+28 -28
wmcs cinder backups: move all backups to 2003 so 2004 can be reimaged	operations/puppet	production	+4 -4
cloudbackups: update partman recipes.	operations/puppet	production	+3 -1
cloudbackup: cram all backups onto cloudbackup1004 so 1003 can be reimaged.	operations/puppet	production	+27 -27
cloudservices1006: use new yaml-based pdns-recursor config	operations/puppet	production	+1 -0
pdns-recursor: use yaml-based config in eqiad1	operations/puppet	production	+1 -2

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Andrew	T375217 Complete upgrading WMCS bare metal hosts to Trixie
Resolved		• taavi	T375082 Upgrade cloudlb hosts to bookworm
Resolved		Andrew	T376277 Reimage cloudweb hosts to trixie
Resolved		jijiki	T292707 ☂ Migrate Wikitech to Kubernetes
Resolved		jijiki	T363125 sustainability of wikitech.wikimedia.org
Resolved		jijiki	T371358 Traffic: Direct wikitech.wikimedia.org to mw-on-k8s
Resolved		jijiki	T371359 Migrate Wikitech's Jobqueue
Resolved		jijiki	T371374 mediawiki-config: consolidate labswiki
Resolved		jijiki	T371608 Memcached: Migrate wikitech to main memcached
Resolved		jijiki	T371537 MVP: Privately serve wikitech via mwdebug1001
Resolved		jijiki	T371360 Apache: Include Wikitech in mw-on-k8s' virtual hosts
Resolved		Ladsgroup	T371588 wikitech self-auth: Allow wikitech to use its own internal authentication
Resolved		jijiki	T371592 LdapAuthentication: Disable extension from Wikitech
Resolved		SLyngshede-WMF	T359820 Developer Account Blocking: Migrate the one-stop Developer (un)Blocking from Wikitech to Bitu
Resolved		SLyngshede-WMF	T376991 Publically available log of blocked/unblocked accounts.
Resolved		jijiki	T373615 Communication for Wikitech/Wikimedia Developer Account migration
Resolved		jijiki	T374114 Review/update wikitech-static syncing after wikitech moves to Kubernetes
Resolved		Andrew	T401899 Reimage cloudgw hosts to Trixie
Resolved		• taavi	T417075 networking.service in failed state on Trixie cloudgw nodes
Open		None	T403154 Upgrade clouddumps hosts to bookworm/trixie
Open	BUG REPORT	None	T391369 If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge
Open		None	T411248 Plan to make clouddumps more resilient and easier to operate
Stalled		None	T413726 upgrade WMCS ceph nodes to Debian Trixie
Duplicate		Andrew	T414346 Upgrade cloudgw nodes to Trixie
Resolved		MoritzMuehlenhoff	T403153 Upgrade cloudcumin hosts to bookworm

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-cloud-feed) [2025-12-04T20:23:53Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-12-04T21:20:15Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Andrew created subtask T413726: upgrade WMCS ceph nodes to Debian Trixie.Jan 4 2026, 6:42 PM

Andrew changed the status of subtask T413726: upgrade WMCS ceph nodes to Debian Trixie from Open to Stalled.

Change #1223268 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudbackup: cram all backups onto cloudbackup1004 so 1003 can be reimaged.

https://gerrit.wikimedia.org/r/1223268

gerritbot added a project: Patch-For-Review.Jan 5 2026, 9:09 PM

Change #1223268 merged by Andrew Bogott:

[operations/puppet@production] cloudbackup: cram all backups onto cloudbackup1004 so 1003 can be reimaged.

https://gerrit.wikimedia.org/r/1223268

Maintenance_bot removed a project: Patch-For-Review.Jan 5 2026, 9:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-05T22:32:17Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-05T22:39:18Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-05T22:49:43Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-05T22:51:08Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-05T22:59:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-05T23:01:57Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-05T23:16:22Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-05T23:18:34Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-05T23:20:57Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T00:54:56Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T00:57:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T01:13:37Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T01:14:33Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T01:18:15Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T01:22:58Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T01:28:53Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T01:33:20Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T01:37:11Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T01:43:49Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T01:52:54Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T02:11:56Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T02:16:32Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T02:20:48Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T02:39:45Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T02:47:07Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T02:58:06Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T03:20:16Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T13:47:45Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T13:54:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T14:39:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-01-06T14:57:34Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T375217)

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudnet1006.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudnet1006.eqiad.wmnet with OS trixie completed:

cloudnet1006 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601061628_andrew_1112267_cloudnet1006.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie executed with errors:

cloudbackup2003 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie executed with errors:

cloudbackup2003 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie executed with errors:

cloudbackup1003 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Change #1224798 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudbackups: update partman recipes.

https://gerrit.wikimedia.org/r/1224798

gerritbot added a project: Patch-For-Review.Jan 8 2026, 8:09 PM

Change #1224798 merged by Andrew Bogott:

[operations/puppet@production] cloudbackups: update partman recipes.

https://gerrit.wikimedia.org/r/1224798

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie executed with errors:

cloudbackup2003 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie executed with errors:

cloudbackup1003 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2026, 8:30 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie executed with errors:

cloudbackup1003 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run failed and the operator aborted
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie executed with errors:

cloudbackup2003 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie executed with errors:

cloudbackup1003 (FAIL)
- Downtimed on Icinga/Alertmanager
- Unable to disable Puppet, the host may have been unreachable
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie executed with errors:

cloudbackup1003 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie executed with errors:

cloudbackup2003 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie executed with errors:

cloudbackup2003 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601082203_andrew_2727225_cloudbackup2003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie completed:

cloudbackup2003 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601090328_andrew_2888147_cloudbackup2003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie executed with errors:

cloudbackup1003 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie completed:

cloudbackup1003 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601091454_andrew_3230765_cloudbackup1003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Change #1224974 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] wmcs cinder backups: move all backups to 2003 so 2004 can be reimaged

https://gerrit.wikimedia.org/r/1224974

Change #1224975 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudbackup: flip all backups from cloudbackup1004 to 1003

https://gerrit.wikimedia.org/r/1224975

Change #1224974 merged by Andrew Bogott:

[operations/puppet@production] wmcs cinder backups: move all backups to 2003 so 2004 can be reimaged

https://gerrit.wikimedia.org/r/1224974

Change #1224975 merged by Andrew Bogott:

[operations/puppet@production] cloudbackup: flip all backups from cloudbackup1004 to 1003

https://gerrit.wikimedia.org/r/1224975

Maintenance_bot removed a project: Patch-For-Review.Jan 9 2026, 3:30 PM

Andrew created subtask T414346: Upgrade cloudgw nodes to Trixie.Jan 12 2026, 2:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudlb1002.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudlb1002.eqiad.wmnet with OS trixie completed:

cloudlb1002 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601121615_andrew_1213428_cloudlb1002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudlb1001.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudlb1001.eqiad.wmnet with OS trixie completed:

cloudlb1001 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601121839_andrew_1283998_cloudlb1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudgw2002-dev.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudgw2002-dev.codfw.wmnet with OS trixie completed:

cloudgw2002-dev (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601141359_andrew_2577069_cloudgw2002-dev.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudgw2003-dev.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudgw2003-dev.codfw.wmnet with OS trixie completed:

cloudgw2003-dev (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601141609_andrew_2640472_cloudgw2003-dev.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup2004.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup2004.codfw.wmnet with OS trixie completed:

cloudbackup2004 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601141749_andrew_2691522_cloudbackup2004.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1004.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1004.eqiad.wmnet with OS trixie executed with errors:

cloudbackup1004 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudbackup1004.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1004.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1004.eqiad.wmnet with OS trixie completed:

cloudbackup1004 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601142043_andrew_2783194_cloudbackup1004.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

MoritzMuehlenhoff closed subtask T403153: Upgrade cloudcumin hosts to bookworm as Resolved.Feb 5 2026, 11:32 AM

• taavi removed a subtask: T403153: Upgrade cloudcumin hosts to bookworm.Feb 5 2026, 2:46 PM

• taavi added a subtask: T403153: Upgrade cloudcumin hosts to bookworm.

Andrew closed subtask T401899: Reimage cloudgw hosts to Trixie as Resolved.Mar 16 2026, 2:36 PM

This is done apart from Ceph hosts, which are tracked elsewhere.

Complete upgrading WMCS bare metal hosts to TrixieClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Complete upgrading WMCS bare metal hosts to Trixie
Closed, ResolvedPublic
Actions

Related Objects
Search...