Task to track reboots for an*, kafka*, etc
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | RKemper | T411568 October 2025 Bullseye reboots: Data Platform Engineering-owned hosts |
Event Timeline
Comment Actions
Change #1214664 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/cookbooks@master] hadoop.reboot-workers: make host override smarter
Comment Actions
an-worker* partially done. made https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1214664 to allow us to reboot a subset of a cluster's hosts while still handling the need to restart one journal node at a time properly. patch needs a bit of fixup.
stat hosts will be restarted tomorrow
Comment Actions
Oh, with respect to the patch, we should also get https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/976163/1/cookbooks/sre/hadoop/reboot-workers.py reviewed and merged at the same time since it's directly relevant to this
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2025-12-04T22:11:08Z] <ryankemper@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on stat[1008-1011].eqiad.wmnet with reason: T411568
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2025-12-04T22:20:44Z] <ryankemper> T411568 Rebooting stat*
Comment Actions
Stat host reboots completed.
Shifting gears to rebooting an-test*. Note there's still lots of an-worker* hosts that need to be rebooted, but that will be easier once I finish the patch I mentioned above.
Comment Actions
Pushed out a patch (https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1214664) to make the rebooting of hadoop workers smarter (namely, we can pass a cumin override and have it just reboot that subset of hosts rather than doing the whole cluster every time - useful if we hit a snag that stops a reboot partway through).
Also added an experimental dry run feature that should hopefully let me verify the new logic is working quickly.
Comment Actions
Resuming an-worker reboots.
Identified 46 hosts needing reboot (kernel < 5.10.244):
an-worker[1117-1125,1127-1140,1143-1147,1149-1150,1152-1160,1168,1187,1193,1198,1200,1206,1208]
Discovered 25 nodes are currently decommissioning (via NameNode UI). Many have "blocks with no live reps" > 0, meaning some blocks exist only on these decommissioning nodes. This means they're unsafe to reboot.
To check this: ssh -L 50470:localhost:50470 an-master1003.eqiad.wmnet, then browse to https://localhost:50470/dfshealth.html and check the Decommissioning Nodes section.
Excluded the 23 decommissioning hosts that overlap with the reboot list:
an-worker[1117-1125,1127-1140]
Proceeding with 23 safe hosts:
an-worker[1143-1147,1149-1150,1152-1160,1168,1187,1193,1198,1200,1206,1208]
Using sre.hadoop.reboot-workers with patch https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1214664 (fixed a bug in patchset 10 where --workers-cumin-query failed when no journal nodes were in the override).
Will revisit the decommissioning hosts once they finish draining.
Comment Actions
Bleh, turned out I'd had a typo in my cumin query, so I'd inverted the hosts: the ones I listed as needing reboot were all already done.
I've done a handful of more hosts (working off the correct host list), but keep hitting landmines with hosts that fail to reboot (usually due to needing to rebuild RAID properly due to a drive that failed).
Decommissioning (25 hosts):
an-worker[1117-1141] — actively draining blocks, will be removed from the cluster.
Current kernel (61 hosts) — 5.10.244+, no reboot needed:
an-worker[1142-1147,1149-1150,1152-1176,1187,1193,1198,1200,1206,1208]
Stale kernel (34 hosts) — need reboot:
an-worker[1148,1151,1177-1186,1188-1192,1194-1197,1199,1201-1205,1207,1209-1236]
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-02-11T02:04:03Z] <ryankemper> [kafka-jumbo]T411568 Rebooting kafka-jumbo one host at a time (sec updates)
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-02-11T04:10:21Z] <ryankemper> [kafka-jumbo] T411568 kafka-jumbo reboots done, and everything looks healthy
Comment Actions
kafka-jumbo successfully completed.
we can resume an-worker, but it's probably best to wait for https://phabricator.wikimedia.org/T411919 to move forward
Comment Actions
@RKemper You don't need to reboot these by hand, BTW. There's a cookbook (sre.kafka.roll-restart-reboot-brokers, which sets proper downtime, reboots one node a time and performs proper cluster health checks before proceeding with the next.
Comment Actions
@MoritzMuehlenhoff oh just to be clear, I did use that cookbook, I think it just doesn't log to this ticket because of the security setting. See https://sal.toolforge.org/log/FS7TSpwBffdvpiTrm1Ne
thanks for looking out!
Comment Actions
Change #1214664 merged by jenkins-bot:
[operations/cookbooks@master] hadoop.reboot-workers: make host override smarter
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-03-17T19:54:05Z] <ryankemper> T411568 rebooted an-test-client1002, an-test-ui1001, an-test-coord1001, an-test-master1001
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-03-17T20:16:12Z] <ryankemper> T411568 rebooted an-test-master1002, an-test-master1003, an-test-master1004, archiva1002
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-03-17T20:38:38Z] <ryankemper> T411568 rebooted an-coord1003, an-coord1004, an-tool1007, an-tool1008, an-tool1011, an-web1001
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-03-17T20:38:44Z] <ryankemper> T411568 failed over HDFS NameNode from an-master1003 to an-master1004, then rebooted an-master1003
Comment Actions
Completed all remaining DPE-owned host reboots today (2026-03-17). All 143 reachable Bullseye hosts are now at or above the 5.10.0-36 target kernel (most are on 5.10.0-39). One host (an-worker1172) remains unreachable and needs to be investigated separately.
Procedure used:
Reboots were done in phases:
- Phase 1 (test cluster): an-test-client1002, an-test-ui1001, an-test-coord1001, an-test-master1001–1004 — rebooted directly, no service failover needed
- Phase 2 (tools/web/coords): an-tool hosts (parallel), an-coord1003 then an-coord1004 (sequential with service checks between), an-web hosts
- Phase 3 (NameNodes): graceful failover of HDFS NameNode from an-master1003→an-master1004 via hdfs haadmin -ns analytics-hadoop -failover, then rebooted an-master1003
What went wrong
I rebooted an-coord1003 without first failing over hive-metastore to an-coord1004. This caused a brief HMS outage during the reboot window. Airflow retries likely covered any affected jobs.
I also didn't post advance notice to Slack before beginning the coord/NameNode phase. I logged SAL/phab entries but didn't properly communicate the scope of what I was doing to the team.
Current cluster state:
- Missing blocks: 0
- Corrupt replica blocks: ~45 (https://grafana.wikimedia.org/d/000000585/hadoop?orgId=1&var-hadoop_cluster=analytics-hadoop&viewPanel=panel-39&from=now-6h&to=now&timezone=utc&var-datasource=000000020&var-worker=$__all) ^ fsck confirms 0 corrupt files so these should self-heal; we'll want to monitor the above grafana panel to see it trend down
- Under-replicated blocks: ~3.26M (actively re-replicating, expected post-reboot)
- Live datanodes: 94
- NameNode: an-master1004 active, an-master1003 standby (failback to 1003 pending)
Outstanding:
- Fail NameNode back to an-master1003 (1003 is up and healthy)
- Investigate an-worker1172 (separate ticket)
- Update T407110 parent checklist
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-03-17T21:38:56Z] <ryankemper> T411568 Failed back HDFS NameNode from an-master1004 to an-master1003; cluster back to original active/standby configuration
Comment Actions
Change #1287290 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/cookbooks@master] hadoop.reboot-workers: drop custom --dry-run
Comment Actions
Change #1287290 merged by jenkins-bot:
[operations/cookbooks@master] hadoop.reboot-workers: drop custom --dry-run