Page MenuHomePhabricator

Toolforge k8s: kube-controller-manager permission errors
Closed, ResolvedPublic

Description

We found this:

aborrero@tools-k8s-control-4:~$ sudo -i kubectl -n kube-system logs kube-controller-manager-tools-k8s-control-4 --timestamps=true
2023-04-10T09:37:00.665673941Z Flag --port has been deprecated, This flag has no effect now and will be removed in v1.24.
2023-04-10T09:37:01.259132291Z I0410 09:37:01.258980       1 serving.go:347] Generated self-signed cert in-memory
2023-04-10T09:37:01.740244861Z I0410 09:37:01.740144       1 controllermanager.go:186] Version: v1.22.17
2023-04-10T09:37:01.750732260Z I0410 09:37:01.750585       1 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
2023-04-10T09:37:01.750849656Z I0410 09:37:01.750585       1 dynamic_cafile_content.go:155] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
2023-04-10T09:37:01.751520088Z I0410 09:37:01.751440       1 secure_serving.go:200] Serving securely on 127.0.0.1:10257
2023-04-10T09:37:01.752979650Z I0410 09:37:01.752928       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
2023-04-10T09:37:01.754822325Z I0410 09:37:01.754770       1 leaderelection.go:248] attempting to acquire leader lease kube-system/kube-controller-manager...
2023-04-10T09:37:05.738137575Z E0410 09:37:05.737990       1 leaderelection.go:330] error retrieving resource lock kube-system/kube-controller-manager: leases.coordination.k8s.io "kube-controller-manager" is forbidden: User "system:kube-controller-manager" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"

Event Timeline

Fixed with a restart similar to the one in T334380#8767907.

taavi claimed this task.