Reimage failed (sre.hosts.reimage exit_code=99) - clear RAID / LVM fix
ActivePublic
Actions

Authored by Dzahn on Fri, Oct 2, 7:43 PM.

Project Tags

None

Referenced Files

	F105876053: Reimage failed (sre.hosts.reimage exit_code=99) - clear RAID / LVM fix
	Fri, Oct 2, 7:43 PM

Subscribers

VRiley-WMF

	# zuul1004.eqiad.wmnet - 2026-10-02
	# Reimage failed (sre.hosts.reimage exit_code=99), install-console showed only a
	# BusyBox shell. The disks still had the software RAID (md0) and LVM (vg0) from
	# the previous install, so partitioning in the installer got stuck.
	# Same issue and fix as on zuul1005 on 2026-09-28.
	#
	# WARNING: this destroys all data on the host's disks.

	# connect to the installer shell (from a cumin host)
	sudo install-console zuul1004.eqiad.wmnet

	# 1. confirm the host and look at the disk layout
	hostname; cat /proc/partitions; lvm pvs
	# -> zuul1004, disks sda sdb sdc sdd with 3 partitions each, md0, VG vg0 on /dev/md0

	# 2. deactivate and remove the LVM volume group
	lvm vgchange -an vg0; lvm vgremove -ff -y vg0
	# -> Logical volume "swap", "root", "srv" successfully removed. Volume group "vg0" successfully removed

	# 3. remove the LVM label from the RAID device
	lvm pvremove -ff -y /dev/md0
	# -> Labels on physical volume "/dev/md0" successfully wiped.

	# 4. stop the RAID array
	mdadm --stop /dev/md0
	# -> mdadm: stopped /dev/md0

	# 5. erase the RAID metadata from the member partitions
	for p in /dev/sd[a-d][1-3]; do echo "$p:"; mdadm --zero-superblock "$p"; done
	# -> "Unrecognised md component device" for sd[a-d]2 (not RAID members), expected

	# 6. wipe the partition tables (start of each disk, and the end where GPT keeps its backup)
	for d in sda sdb sdc sdd; do echo "wiping /dev/$d"; dd if=/dev/zero of=/dev/$d bs=1M count=16; sz=$(cat /sys/block/$d/size); dd if=/dev/zero of=/dev/$d bs=512 seek=$((sz - 32768)) count=32768; done; sync

	# 7. abort the waiting cookbook, then reimage again (from a cumin host)
	sudo cookbook sre.hosts.reimage --os trixie -t T427353 zuul1004

Event Timeline

Dzahn created this paste.Fri, Oct 2, 7:43 PM

Dzahn added a subscriber: VRiley-WMF.

Dzahn mentioned this in T427353: Repurpose ganeti102[3456] for Zuul migration.Fri, Oct 2, 7:47 PM

Reimage failed (sre.hosts.reimage exit_code=99) - clear RAID / LVM fix ActivePublicActions

Event Timeline

Reimage failed (sre.hosts.reimage exit_code=99) - clear RAID / LVM fix
ActivePublic
Actions