Page MenuHomePhabricator

Find/ban "ghost" Elastic hosts
Closed, ResolvedPublic

Description

Per parent ticket, several Elastic hosts have failed to reimage. Most of the time when this happens, the hosts come back up and rejoin the cluster.

This is causing the cluster to stay in yellow status, because any primary shard hosted on these "ghost" hosts cannot replicate to OpenSearch hosts.

Banning the hosts should force the primary shards onto OpenSearch (cirrussearch) hosts, allowing the cluster to return to green status.

Creating this ticket to:

  • Find and ban all elastic hosts
  • Force shards to be redistributed
  • Confirm operation

Event Timeline

bking changed the task status from Open to In Progress.May 27 2025, 1:58 PM
bking triaged this task as Medium priority.

Found and banned the following hosts:

  • elastic1054
  • elastic1055
  • elastic1056
  • elastic1058
  • elastic1059
  • elastic1063
  • elastic1067
  • elastic1110

I've banned the hosts and forced a reroute. The cluster state is green, so this ticket can be closed. Work to decom the old hosts continues in T394350...