While working on T215272: Upgrade Prometheus to 2.7 in deployment-prep and tools I noticed tools-prometheus hosts can't talk to tools-worker hosts on tcp port 10250: https://tools-prometheus.wmflabs.org/tools/targets#job-k8s-node this is most likely due to ferm rules on tools-worker hosts not allowing kubelet connections from prometheus hosts.