Can this task be closed? By default hosts reimage now but they do keep /srv (T251768)

From the perspective of DBA, this issue is mostly resolved. Most DB machines will keep /srv on reimage by default (the exceptions are in T255768: Create reuse recipes for tendril/zarcillo/dbprov/backup hosts). The same needs to be done for the backup machines.

As the general issue isn't resolved, i'm adding this to the SRE-tools tag as discussed with @Volans for future work.

https://phabricator.wikimedia.org/T277007 was a recent case where reimaging the OS (after a BIOS update) was problematic for DBA as well.

@LSobanski I'll try to give you some context from the SRE I/F team side of things. Any feedback will be greatly appreciated, also to help set the team's priority around those items.

Current status

• We've noticed that sometimes the PXE next boot override setting is not super reliable we had both cases of:
  • Hosts where setting force PXE via remote IPMI succeeded, checking the override returned force PXE but then the host rebooted into the local system
  • Hosts where after a force PXE reboot the force setting was not reset back to non-force after a reboot
• There have been also cases in which the default boot was set to PXE at the BIOS level (as opposed to just force next boot to PXE via IPMI)
• Monitoring of the BMC is kept to a minimal because we've seen issues in the past when pinging too often the BMC via remote IPMI or SSH for monitoring purposes that lead to failures, making them either noisy or unreliable in the moment of need.

What has been done in the past

• I did an audit on remote IPMI in 2018 (see T193155) where, among other things, I found ~150 hosts had force PXE set. And then fixed all the outstanding issues that the audit found
• Thanks to @Kormat's work the partman recipe allow to save the local data on accidental reimage, when used
• The reimage script checks both that the force PXE is set at reimage time before rebooting and that it get reset after the reboot, alerting the user with a warning, that I fear is mostly ignored because not very clear.

Short term possible actions

• Perform a new audit of the whole fleet to see the current status and asses the current situation
• Improve the reimage script check so that it fails more clearly if it detects the force PXE bit after the reimage is done and/or try to reset it multiple times.

Long term solutions

• There is a work in progress project to automate all the basic BIOS settings of DELL's hosts that should be live by end of this fiscal (end of June) and that would allow to prevent all cases of manual misconfiguration where the BIOS setting was left to boot from PXE instead of local disks
• There is a plan (no ETA on this yet) to create a PXE boot menu that will have various functionalities like reimage, wipe, live-os, etc.. and that will default to boot from local disks. When this will be available the idea is to have all hosts boot by default on PXE that will then default to local disks and of course have the BIOS fallback to local disks in case PXE is unreachable

If there is some agreement on the short term actions I could take care of both of them in the next week or two.

@Volans thanks for the explanation!
There's something I have been wondering for a long time, not sure if this fits on this part of the project or if it needs to be addressed somewhere else, anyways, posting it here.
Is it doable at some point to also upgrade firmware/bios to the latest available when reimagining a host? We've found many hosts that come with a very old BIOS version when sent from factory and despite having a newer version available, that's not installed. Is that something the installation script could handle?

Thanks!

@Marostegui yes, I didn't mention all the other efforts because a bit off topic, but firmware upgrade is too in scope and in our roadmap. And totally makes sense once we have that to integrate it into the reimage workflow so that we naturally upgrade them along with OS upgrades.

@Volans Both the short and long term actions make sense to me, thanks for the summary.

Removing SRE, has already been triaged to a more specific SRE subteam

As an update, with the current situation even if a host reboots into PXE the DHCP will not provide any IP so the reimage will NOT happen. The DHCP is now dynamic and set by the reimage/dhcp cookbooks on the fly just during the time of the reimage.
The only current risk is if a dhcp snippet is left by the reimage/dhcp cookbooks (because ctrl+C-ed twice or killed -9) and that host does reboot into PXE, but this is easily fixable in various ways deleting those potentially leftover files.
So I think that the original concern has been almost completely removed.

Agreed, I think we can simply resolve task.

So I think that the original concern has been almost completely removed.

Agreed, I think we can simply resolve task.

+1