Change 849595 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] R:swift::label_filesystem: jst check that any lable is on the disk

https://gerrit.wikimedia.org/r/849595

as currently set up, puppet is unhappy if the SSDs come up as anything other than sda and sdb

I suspect that is related to the following

the label renaming aux drives i.e. https://phabricator.wikimedia.org/T308677#8340723

this change could make that a bit better but its not a great fix as its just looking for *a* label i.e. any label.

The underlining issues is that theses classes try to make the disk mounting stable by adding a label, but the label is based on the thing that is unstable to begin with i.e. the device name. We need to mount theses disks with something that doesn't change on every boot. Currently with the rust disks this works because we only ever add the label once, after that if a label exists no matter what it is we dont relabel. (see issue highlighted by the audit cookbook) however as mentioned above for the ssd disks we relabel them every time they change which means there is no benefit to labelling them in the first place other then adding a bit if confusion.

We can continue to mount things in /srv/swift-disks/sd* using something more stable then a label based on the device name i.e.

/dev/disk/by-path/pci-0000:3b:00.0-scsi-0:0:12:0-part3 - /srv/swift-storage/sda3
/dev/disk/by-path/pci-0000:3b:00.0-scsi-0:0:13:0-part3 - /srv/swift-storage/sdb3
/dev/disk/by-path/pci-0000:3b:00.0-scsi-0:0:12:0-part4 - /srv/swift-storage/sda4
/dev/disk/by-path/pci-0000:3b:00.0-scsi-0:0:13:0-part4 - /srv/swift-storage/sdb4

but i honestly dont see why we would proliferate this as its only going to cause confusion, when after a few weeks, months years of reboots /dev/disk/by-path/pci-0000:3b:00.0-scsi-0:0:12:0-part3 is lablked as /dev/sdn3 but still mounted as /srv/swift-storage/sda3 and .devsda3 dosn't even exist (again see audit cookbook results)

I also note here that as far as i can tell the device names them self i.e. /dev/sda can't be made stable. the kernel scans and names them as they show up, then udev quires the disks ( with udevadm info /dev/sda) and creates the stable names under /dev/disks

In T308677#8339622, @jbond wrote:

luckily puppet doesn't relabel them

I have noticed for the aux drives which use swift::lable_filesystem that they do relabel the disks, this means you could get some really strange behaviour if dev/sda came up as /dev/sdb or the other way round

Sorry, to come back to this, a nit: puppet tries to relabel them but fails (because xfs_admin -L won't adjust a mounted partition I think). So the failure mode is usually puppet unhappyness rather than too much relabelling. [I don't think this really affects your main point, but seemed worth noting in passing]

...so I think your change might at least get us back to systems booting reliably, which would be quite a win!

Change 849595 merged by Jbond:

[operations/puppet@production] R:swift::label_filesystem: jst check that any lable is on the disk

https://gerrit.wikimedia.org/r/849595

In T308677#8290550, @jbond wrote:

just putting a note here. after looking at the dell anisble module it seems we should be able to change the disk (e.g. /redfish/v1/Systems/System.Embedded.1/Storage/RAID.Integrated.1-1/Drives/Disk.Bay.25:Enclosure.Internal.0-2:RAID.Integrated.1-1) ["Oem"]["Dell"]["DellPhysicalDisk"]["RaidStatus"] = 'Offline'

or possibly

'post', "/redfish/v1/Systems/System.Embedded.1/Oem/Dell/DellRaidService/Actions/DellRaidService.ChangePDState
 data={"TargetFQDD": drive_id, "State": Offline}

this unfortunately doesn't work as I expected

Bother :-/

Change 848451 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] C:swift: add swift disks fact

https://gerrit.wikimedia.org/r/848451

Change 848418 merged by Jbond:

[operations/puppet@production] C:swift::storage: add variable for data directory

https://gerrit.wikimedia.org/r/848418

Change 848451 merged by Jbond:

[operations/puppet@production] C:swift: add swift disks fact

https://gerrit.wikimedia.org/r/848451

Change 848419 merged by Jbond:

[operations/puppet@production] P:swift::storage: add new resource to format via pci path

https://gerrit.wikimedia.org/r/848419

Change 848420 merged by Jbond:

[operations/puppet@production] ms-be2050: enable disks by path configuerations

https://gerrit.wikimedia.org/r/848420

Change 859581 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] swift::mount_filesystem: allow overriding the mount point

https://gerrit.wikimedia.org/r/859581

Change 859581 merged by Jbond:

[operations/puppet@production] swift::mount_filesystem: allow overriding the mount point

https://gerrit.wikimedia.org/r/859581

Change 859584 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] swift: Allow for mounting using the device directly

https://gerrit.wikimedia.org/r/859584

Change 859584 merged by Jbond:

[operations/puppet@production] swift: Allow for mounting using the device directly

https://gerrit.wikimedia.org/r/859584

Change 859592 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] swift: move ms-be2050 to new naming schema

https://gerrit.wikimedia.org/r/859592

Change 859607 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] install_server: Add dynamic raid configuration

https://gerrit.wikimedia.org/r/859607

Change 859992 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] swift: base the object number on the scsi path

https://gerrit.wikimedia.org/r/859992

Change 859992 merged by Jbond:

[operations/puppet@production] swift: base the object number on the scsi path

https://gerrit.wikimedia.org/r/859992

In T308677#8346238, @jbond wrote:

The underlining issues is that theses classes try to make the disk mounting stable by adding a label, but the label is based on the thing that is unstable to begin with i.e. the device name. We need to mount theses disks with something that doesn't change on every boot.

I'm probably missing something here, but why don't we simply mount the disks based on UUIDs? blkid will print that and it's stable across reboots, e.g. for the disk I'm using as my root partition:

$ sudo blkid /dev/nvme0n1p2
/dev/nvme0n1p2: UUID="1586da60-3648-4ecf-ae57-1acdbde90080" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="6924d437-fff4-47b9-bc94-696c35a7843d"

And then it /etc/fstab:

UUID=1586da60-3648-4ecf-ae57-1acdbde90080 /               ext4    errors=remount-ro 0       1

Change 859607 merged by Jbond:

[operations/puppet@production] install_server: Add dynamic raid configuration

https://gerrit.wikimedia.org/r/859607

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

In T308677#8417608, @MoritzMuehlenhoff wrote:

In T308677#8346238, @jbond wrote:

The underlining issues is that theses classes try to make the disk mounting stable by adding a label, but the label is based on the thing that is unstable to begin with i.e. the device name. We need to mount theses disks with something that doesn't change on every boot.

I'm probably missing something here, but why don't we simply mount the disks based on UUIDs? blkid will print that and it's stable across reboots, e.g. for the disk I'm using as my root partition:

We could use UUID's however the problems is about how we manage that i.e. we would need a way to first profile the the server so that we no which uuid relate to the rust fdisks and which to the ssd disks and then plug them into hiera somewhere. Further when replacing a disk we would also need to update the hiera data. Using the scsi by path value makes it a little easier to:

• handle replaced disks as they will have the same bath
• map the disk to a specific mount point e.g. container1 as we can use the scsi path value as the mount point e.g. 0:0:1:0 -> /mnt/object1, 0:0:7:0 -> /mnt/object7 etc

I have now update ms-be2050 to use the newer mounting scheme based on the by-path id and that seems to be able to reboot and apply its puppet policy without issue.

I have also made some progress towards the re-imaging process in that i can use a script to discover the ssd disks and plug them into partman. This mostly seems to work in that d-i sees the configuration partition table, formats the xfs partitions 3 and 4 and attempts to create the software raid drive allusing the correct disks . however i seems to hit an issue with the last step and d-i stops with the following error

	Nov 24 11:39:01 partman-auto-raid: mdadm: cannot open /dev/sdl: Device or resource busy
	Nov 24 11:39:01 partman-auto-raid: Error creating array /dev/md1

This seems to be a more generic issue with partman creating the sowftare raid device then ms-be specific, and thus seems to be my time at the WMF where i need to dig into partman internals :/. but if anyone else has hit a similar issue or has some pointers please let me know

This seems to be a more generic issue with partman creating the sowftare raid device then ms-be specific, and thus seems to be my time at the WMF where i need to dig into partman internals :/. but if anyone else has hit a similar issue or has some pointers please let me know

I just reliaed my partman-auto-raid/recipe uses /dev/sda#/dev/sdb vs /dev/sda1#/dev/sdb1 im going to try this change and see if it works

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye completed:

• ms-be2050 (WARN)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202211241242_jbond_583391_ms-be2050.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye completed:

• ms-be2050 (WARN)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202211241413_jbond_600747_ms-be2050.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Change 860581 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] install_server: migrate ms-bs_simple top GPT

https://gerrit.wikimedia.org/r/860581

ok with T308677#8419843 and T308677#8420119 i have now managed to successfully re-image ms-be2050 a couple of times with no problems. I think this resolves the main issue of the task, although ill leave it for @MatthewVernon to confirm.

There where two issues with theses hosts.

The first was that reimages where unstable as partman recipe uses sda and sdb regardless of weather theses disks where ssd or spinning disks. To fix this i have created a partman_early script to find the SSD disks and use them to construct the raid devices.

The second issues is that all of the swift ring mount points where mounted under /srv/swift-storage/sd? this cases issues when disks come up in the wrong order specifically if one of the ssd disks comes up with the wrong name. To fix this i have migrated away from using labels and the sd? device name to using the /dev/disk/by-path udev maps. This provides a much more stable mounting profile and also has the benefit that a replaced disk will be mounted in the place (as it will have the same scisi path).

Fixing theses two issues should provide for a much smoother experience, however all servers will need to be migrated to the new schema. From a data PoV it would be possible to mount both disks in both locations while we update the various swift configurations. however i suspect that things will be a bit tricker then that as we will somehow have to tell swift that ms-be250:/srv/swift-storage/sdc1 is now ms-be250:/srv/swift-storage/objects1. This is made more complicated by the fact that the mappings wont be the same across servers i.e. ms-be2049 may have /srv/swift-storage/sdc1 => /srv/swift-storage/objects8. As such i suspect migrating to this scheme will require a full rebuild of all ms-be servers (but again ill wait for @MatthewVernon to confirm)

If a rebuild of all back-ends is required then i think we should also consider converting the partitioning to use GPT and also move to having two swap partitions instead of having on mirrored swap partition. i attempted this in T308677#8419864 and T308677#8419970, hence the two failures, using a patch similar to 860581 however i was unsuccessful. The main issue is that we need to add an additional boot partition which means that the xfs partitions need to become extended instead of primary and im not sure i got the syntax correct. moving to this would also need a minor change to how we identify the account and container disks however that is not concerting.

The last point i wanted to raise relates to T309027 where disks where presenting as rotational and not SSD. During that task it was discussed to use the convert-ssds cookbook to migrate all disks away from the current schema where all disks are configured as a single raid0 devices with one HD present. To a new schema where we just configure all disks as none-raid disks. However looking at the current servers i noticed that most (possibly all) are still configured in the old manner. As things now work i guess this is not such a big issue however i think we should agree on what format as this discussion impacts the by-path value that the disk gets. My vote would be to convert everything to a None raid disk (while we re-image) as it removes the dell virtual drive and simplifies things slightly

ms-be2050 looks good to me now, thank you :)

I think any approach other than re-imaging is going to be fiddly at best; you'd have to reconstruct the sdX -> objectY mapping at boot each time before the filesystems were mounted, I think. And I don't think there's any "rename" operation in swift-ring-builder, so I think you're still going to have to drain (or just drop) the old names and then add the new ones (maybe with the immediate setting).

I think what we did was convert all the SSDs to non-RAID but left the spinning rust alone. I don't know if there's any performance gain to having them be single RAID0 devices (there's certainly admin pain of having to map between the RAID device and physical one).

In T308677#8420280, @MatthewVernon wrote:

ms-be2050 looks good to me now, thank you :)

Great

I think any approach other than re-imaging is going to be fiddly at best; you'd have to reconstruct the sdX -> objectY mapping at boot each time before the filesystems were mounted, I think. And I don't think there's any "rename" operation in swift-ring-builder, so I think you're still going to have to drain (or just drop) the old names and then add the new ones (maybe with the immediate setting).

Thats what i assumed

I think what we did was convert all the SSDs to non-RAID but left the spinning rust alone.

Ahh that explains things

I don't know if there's any performance gain to having them be single RAID0 devices (there's certainly admin pain of having to map between the RAID device and physical one).

Tbh i doubt there would be a performance gain if anything i would expect some tiny overhead (we have already seen some issues with the ssds) but who knows perhaps you can tweak different cache and seek settings if its virtual. however id say that the admin points could end up being a bit painful. For instance right now the scis 0:0:0:0 physical disk shows up as 0:2:2:0 im guessing it is starting at index 2 instead of 0 because originally the ssd where mapped to the virtual disks 0 and 1 (probably to try and win the pci scan race). however this shows that its a little bit less deterministic for instance if two disks died they may get configured with a different virtual disk number. its possible the next virtual disk will be assigned 0:2:0:0 for instance. So id say if we are going to re-image anyway it probably worth fixing the other cookbook* to make all disks none raid and then call the reimage cookbook?

*as this is a one of task we doesn't need to be as gentle as riccardo was trying to be :). We can then add something to the provisioning cookbook to make sure new hosts are configured in the same way

what do you think about the GPT/swap recommendations?

Yes, I think I agree; although I'm not sure if Riccardo managed to iron out all the issues with the convert-ssd cookbook in the end (I did a bunch of them manually whilst reimaging anyway).

I think not RAIDing the swap makes a deal of sense, and am happy with GPT partition tables.

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye completed:

• ms-be2050 (WARN)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202211281327_jbond_1524484_ms-be2050.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run failed, asking the operator what to do
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Change 860581 merged by Jbond:

[operations/puppet@production] install_server: migrate ms-bs_simple top GPT

https://gerrit.wikimedia.org/r/860581

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye executed with errors:

• ms-be2050 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run failed, asking the operator what to do
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin2002 for host ms-be2050.codfw.wmnet with OS bullseye completed:

• ms-be2050 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Unable to disable Puppet, the host may have been unreachable
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202211281721_jbond_1563631_ms-be2050.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Change 859470 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/cookbooks@master] convrt-ssds: update cookbook to reimage ms-be with new partition schema

https://gerrit.wikimedia.org/r/859470

Change 875811 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: Remove ms-be2050 from the rings

https://gerrit.wikimedia.org/r/875811

Change 875811 merged by MVernon:

[operations/puppet@production] swift: Remove ms-be2050 from the rings

https://gerrit.wikimedia.org/r/875811

Change 859592 merged by MVernon:

[operations/puppet@production] swift: move ms-be2050 to new naming schema

https://gerrit.wikimedia.org/r/859592

Change 877101 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] hiera: remove ms-be2050 from servers_per_port 0 setting

https://gerrit.wikimedia.org/r/877101

Change 877101 merged by MVernon:

[operations/puppet@production] hiera: remove ms-be2050 from servers_per_port 0 setting

https://gerrit.wikimedia.org/r/877101

Change 894009 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] install_server: use newer partman setup for new ms backends

https://gerrit.wikimedia.org/r/894009

Change 894009 merged by MVernon:

[operations/puppet@production] install_server: use newer partman setup for new ms backends

https://gerrit.wikimedia.org/r/894009

Change 895141 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] hiera: use a regex to specify new-style storage hosts

https://gerrit.wikimedia.org/r/895141

Change 895141 merged by MVernon:

[operations/puppet@production] hiera: use a regex to specify new-style storage hosts

https://gerrit.wikimedia.org/r/895141

Mentioned in SAL (#wikimedia-operations) [2024-02-06T14:32:31Z] <Emperor> debug convert-disks cookbook against out-of-use ms-be2044 T308677

Change 859470 merged by jenkins-bot:

[operations/cookbooks@master] convert-disks: update cookbook to reimage ms-be with new partition schema

https://gerrit.wikimedia.org/r/859470