The ongoing OpenSearch migration (T388610) requires a lot of changes to our load balancer pools.
We have accidentally triggered alerts by forgetting to depool nodes before a reimage. We might also have hosts failing health checks because they haven't properly joined the cluster, or for any number of reasons. A real-time view of each node's health status would help us triage these issues quickly, hopefully before they generate alerts for other teams.
Creating this ticket to:
[] Find the best source of this information (@Volans pointed me to [[ https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DPybalBackendDown | this alerts dashboard ]])
[] Use the information to create a tool (probably a dashboard) that displays real-time (or as near as is practical) information on node status per pool.