This achieves several goals, test and ensure that the hosts can reboot without issuse (kernel/grub/etc.); make sure that the cluster can lose one host without any issues; exercise our skillset on moving aronud pieces of the cluster. My goal is to do this periodically, and very likely automate most of it.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | dcaro | T272458 [ceph] Do a periodical all round machine reboot | |||
| Resolved | dcaro | T272993 [ceph] investigate osd crash in eqiad |
Event Timeline
Comment Actions
Great idea. Once we find success with this approach with ceph based nodes, I would encourage a similar cadence to apply to HV's and other machines we operates.
Future looking, I'm curious about maintenance operations occurring during this scheduled downtime. I'll note this is beyond scope for this and very future looking. However, what about canary testing updates as part of this process? AKA, can we update packages, OS, kernel or firmware during this downtime and reintroduce them to the cluster without ill effect?