We had this in the past already and it kind of healed itself IIRC but we now have the situation again that staging-eqiad do not show up in Elasticsearch/Kibana at all.
Last log lines in elasticsearch:
Aug 20, 2021 @ 10:43:39.207 kubestage1002.eqiad.wmnet Aug 20, 2021 @ 10:43:39.193 kubestage1001.eqiad.wmnet
/var/log/syslog.6.gz:Aug 20 10:43:39 kubestage1002 rsyslogd: mmkubernetes: failed to connect to [https://kubestagemaster.svc.eqiad.wmnet:6443/api/v1/namespaces/kube-system/pods/coredns-6674d59bd8-dhl56] - 7:Couldn't connect to server [v8.1901.0] /var/log/syslog.6.gz:Aug 20 10:43:39 kubestage1002 rsyslogd: action 'action-0-mmkubernetes' (module 'mmkubernetes') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0]
It turns out the mmkubernetes plugin is terminated rather than suspended in case of connection errors to k8s API. So a retry would never happen: https://github.com/rsyslog/rsyslog/issues/4669