Page MenuHomePhabricator

Errors from k8s API for user 'prometheus'
Closed, ResolvedPublic

Description

Noticed this in /var/log/prometheus/server.log, looks like the k8s API is returning 403 for (some?) calls. Note both k8s and k8s-staging Prometheus are affected, not sure why though, cc @akosiaris

Nov 15 14:14:13 prometheus1003 prometheus@k8s[4965]: level=error ts=2019-11-15T14:14:13.763785856Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:372: Failed to list *v1.Node: nodes is forbidden: User \"prometheus\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"
Nov 15 14:14:14 prometheus1003 prometheus@k8s-staging[8278]: level=error ts=2019-11-15T14:14:14.216718797Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:372: Failed to list *v1.Node: nodes is forbidden: User \"prometheus\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"
Nov 15 14:14:14 prometheus1003 prometheus@k8s[4965]: level=error ts=2019-11-15T14:14:14.766020612Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:372: Failed to list *v1.Node: nodes is forbidden: User \"prometheus\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"

Details

Related Gerrit Patches:
operations/deployment-charts : masterRBAC: Allow prometheus access to nodes resources

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, Nov 15, 2:54 PM
akosiaris triaged this task as Low priority.Fri, Nov 15, 5:20 PM

I think we can solve this by just adding system:heapster role to our prometheus kubernetes group. I 'll try that approach but for now I am lowering to Low as it isn't causing an issue with metrics gathering (at least the metrics we currently rely on)

Change 551266 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] RBAC: Allow prometheus access to nodes resources

https://gerrit.wikimedia.org/r/551266

I think we can solve this by just adding system:heapster role to our prometheus kubernetes group. I 'll try that approach but for now I am lowering to Low as it isn't causing an issue with metrics gathering (at least the metrics we currently rely on)

Sounds good to me! Thanks for taking a look

Change 551266 merged by jenkins-bot:
[operations/deployment-charts@master] RBAC: Allow prometheus access to nodes resources

https://gerrit.wikimedia.org/r/551266

akosiaris closed this task as Resolved.Tue, Dec 3, 11:21 AM
akosiaris claimed this task.

Problem fixed by the change above