Change Details

In incident {T367348} we discovered a failure case where the Kubernetes API server's port is open but it's not responding to any API requests. The current HAProxy health check is a simple TCP check, I think it should be replaced by a HTTPS check for the `/healthz` endpoint. Also review that we have alerts for the case where the API servers are not healthy according to HAProxy.

In incident {T367348} we discovered a failure case where the Kubernetes API server's port is open but it's not responding to any API requests. In that case HAProxy still sees the failed servers as up and tries to send traffic to them. The current HAProxy health check is a simple TCP check, I think it should be replaced by a HTTPS check for the `/healthz` endpoint. Also review that we have alerts for the case where the API servers are not healthy according to HAProxy.

In incident {T367348} we discovered a failure case where the Kubernetes API server's port is open but it's not responding to any API requests. In that case HAProxy still sees the failed servers as up and tries to send traffic to them. The current HAProxy health check is a simple TCP check, I think it should be replaced by a HTTPS check for the `/healthz` endpoint. Also review that we have alerts for the case where the API servers are not healthy according to HAProxy.