Although our hosts are technically on OpenSearch now, we still use sre.elasticsearch.rolling-operation to restart the fleet. The cookbook has an argument called --start-datetime we can use to recover from a failed cookbook run without having to restart services on the hosts that were already successfully restarted.
Unfortunately, the rolling-operation cookbook uses the JVM start time for --start-datetime , which only works for service restarts. It doesn't account for host reboots, which are also needed occasionally (ref T407110 amongst others).
Creating this ticket to:
- Adjust the cookbook and/or spicerack logic to handle reboots correctly.
- Verify operation