Page MenuHomePhabricator

tools-prometheus can't connect to tools-worker-* on port 10255 for Kubernetes metrics
Closed, ResolvedPublic

Description

modules/role/manifests/toollabs/k8s/worker.pp allows $master_hosts_ferm to connect to port 10255, which isn't right.

Due to this, Prometheus-Labs is missing statistics about k8s workers.

Event Timeline

Change 486142 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/puppet@production] toollabs::k8s::worker - Allow prometheus to access read-only metrics port 10255

https://gerrit.wikimedia.org/r/486142

Change 486142 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/puppet@production] toollabs::k8s::worker - Allow prometheus to access read-only metrics port 10255

https://gerrit.wikimedia.org/r/486142

GTirloni updated the task description. (Show Details)

Change 486142 abandoned by GTirloni:
toollabs::k8s::worker - Allow prometheus to access read-only metrics port 10255

Reason:
Linter won't allow fix without refactoring role into role/profile. Toolforge k8s is to be refactored soon in another task.

https://gerrit.wikimedia.org/r/486142

@Bstorm I'm pretty sure this was fixed recently, but I can't find the bug/patch that fixed it. Do you remember how it was done? The data in tools-prometheus.wmflabs.org seems to start around 2019-07-23T03:25.

Bstorm claimed this task.

Yup! I fixed this but I didn't realize there was a ticket around.