Summary

Write a Spicerack cookbook (sre.hadoop.reboot-coordinators) that safely reboots analytics coordinator hosts (an-coord1003, an-coord1004) by verifying service failover before proceeding.

The failover infrastructure is already in place (T280905): both coordinators run HMS, HiveServer2, and Presto; DNS CNAMEs (analytics-hive.eqiad.wmnet, analytics-presto.eqiad.wmnet) exist for traffic routing; Kerberos keytabs and TLS certs cover both hosts. What's missing is automation to enforce the correct operational sequence.

Motivation

During the T411568 reboot campaign (2026-03-17), an-coord1003 was rebooted without first failing over Hive Metastore, causing a brief HMS outage. This was a manual error — I forgot the failover step. A cookbook would prevent this class of mistake by refusing to reboot a coordinator that is still serving traffic via DNS.

Previously, btullis performed the correct procedure manually during an SSD firmware update (T394499): merge a DNS change in operations/dns to redirect CNAMEs to the standby coordinator, then proceed with maintenance. This cookbook would codify and partially automate that procedure.

Proposed behavior

Pre-reboot checks:
1. Resolve analytics-hive.eqiad.wmnet and analytics-presto.eqiad.wmnet
2. If either CNAME points to the target host:
   a. Warn the operator and list which CNAMEs need updating
   b. Verify HMS (port 9083) and Presto (port 8281) are healthy
      on the *other* coordinator
   c. Pause and instruct operator to merge DNS change in operations/dns and deploy via authdns-update
   d. After operator confirms, re-resolve CNAMEs to verify propagation
   e. Abort if DNS still points to the target host
3. If neither CNAME points to the target host, proceed directly

Reboot:
4. Set Icinga/Alertmanager downtime
5. Reboot the host (via sre.hosts.reboot-single or equivalent)
6. Wait for host to come back

Post-reboot verification:
7. Verify hive-metastore, hive-server2, presto services are running
8. Check Presto active worker count matches expected value
9. Remove downtime

Note: downtime (step 4) must persist through post-reboot verification
(step 8), so we may need to manage downtime ourselves rather than
delegating to sre.hosts.reboot-single (which lifts downtime on
host recovery). Implementation detail to sort out during development.

The DNS change itself remains a manual Gerrit merge + authdns-update step. The cookbook's value is in the guard rails: it won't let you reboot a coordinator that's still serving traffic, and it verifies DNS propagation before proceeding.

Future improvements (out of current scope)

• Automate the DNS change directly from the cookbook (commit to operations/dns, merge, run authdns-update)