Page MenuHomePhabricator

Validate that maintain-kubeusers works with k8s 1.18
Closed, ResolvedPublic

Description

Usually, you'd spin up a cluster with the version of k8s on minikube for this task and ensure that the tests pass, following the instructions in the README for https://gerrit.wikimedia.org/g/labs/tools/maintain-kubeusers/+/refs/heads/master

If they don't pass, you'll need to check the errors and changelogs for 1.18 to find out why. It's usually api version changes, but it can be the python-kubernetes client getting too old (it is out of date as is).

If it cannot be run on 1.18 because it is too old, you can validate against 1.19 usually since anything that works on 1.19 should work on 1.18. Deploying to toolsbeta and creating a new tool there (which is only possible for WMCS staff at this time) is the final confirmation before building the image with the :latest tag and deploying to tools.

Event Timeline

Minikube is refusing to start a 1.18 env for me so I'm testing with 1.19. tests/test_api.py is failing all tests, certificatesigningrequests.certificates.k8s.io \"tool-blurp\" is forbidden: user not permitted to approve requests with signerName \"kubernetes.io/legacy-unknown\". Kubernetes has the concept of a "certificate signer" and starting with 1.18 the service account approving the CSR must have access to use that specific approver, we don't grant that. Kubernetes 1.19 will introduce the stable v1 of the certificates API which disallows the use of the default "kubernetes.io/legacy-unknown" signer. 1.18 still supports the legacy one, but given the new ones are already available, I believe kubernetes.io/kube-apiserver-client is correct for these client certs.

Change 680244 had a related patch set uploaded (by Majavah; author: Majavah):

[labs/tools/maintain-kubeusers@master] Add RBAC rules to allow using certificate signers

https://gerrit.wikimedia.org/r/680244

It won't be as easy as I would have hoped for, mostly because the Python k8s library is behind k8s releases and does not support some of the features that I would want it to, namely specifying the certificate signer or the stable certificates/v1 api. The current setup (see patch) will work until 1.22.

In T280300#7008122, @Majavah wrote:

Minikube is refusing to start a 1.18 env for me so I'm testing with 1.19. tests/test_api.py is failing all tests, certificatesigningrequests.certificates.k8s.io \"tool-blurp\" is forbidden: user not permitted to approve requests with signerName \"kubernetes.io/legacy-unknown\". Kubernetes has the concept of a "certificate signer" and starting with 1.18 the service account approving the CSR must have access to use that specific approver, we don't grant that. Kubernetes 1.19 will introduce the stable v1 of the certificates API which disallows the use of the default "kubernetes.io/legacy-unknown" signer. 1.18 still supports the legacy one, but given the new ones are already available, I believe kubernetes.io/kube-apiserver-client is correct for these client certs.

Great catch!

In T280300#7008144, @Majavah wrote:

It won't be as easy as I would have hoped for, mostly because the Python k8s library is behind k8s releases and does not support some of the features that I would want it to, namely specifying the certificate signer or the stable certificates/v1 api. The current setup (see patch) will work until 1.22.

Yes, the python client is lagging behind badly. This may need to be rewritten in golang if that keeps up.

taavi triaged this task as Medium priority.Apr 17 2021, 9:03 AM

Change 680244 merged by jenkins-bot:

[labs/tools/maintain-kubeusers@master] Add RBAC rules to allow using certificate signers

https://gerrit.wikimedia.org/r/680244

Mentioned in SAL (#wikimedia-cloud) [2021-04-20T21:49:57Z] <bstorm> tagged the latest maintain-kubeusers and deployed to toolforge (with kustomize changes to rbac) after testing in toolsbeta T280300

Done by creating a new tool test7 on toolsbeta.