Today during a seemingly unrelated issue (high number of unallocated shards on the eqiad search cluster) the production logstash cluster went into an unknown degraded/broken state until the eqiad search cluster issue was address. This seems to highlight a cross-dependency between elasticserach clusters that under certain failure modes could leave us blind while debugging a production issue, cause gaps in logs, etc.

Creating a task to discuss further and investigate possible solutions.