Page MenuHomePhabricator

Move kubernetes authentication to using X.509 client certs
Open, LowPublic


Currently we use tokenauth for authenticating clients to kubernetes servers, which works but requires share state between all the k8s master when running in HA configuration. This 'state' is the tokenauth csv file, and updating this file also requires a k8s apiserver restart, which is not nice.

Instead, we should use X.509 client certs for authentications. We can have a CA setup just for this, and it can be used to sign certs. This way we only need to share the CA between the k8s masters. ABAC info also needs to be shared, but that can be independently generated on each node, I think.

Event Timeline

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptAug 29 2016, 7:30 AM
scfc added a subscriber: scfc.Nov 27 2016, 2:21 AM

I had thought about using certificates to authenticate tools in services in the past, mostly so that there would be only one "thing" per tool and not a separate pair of username and password for each service there is.

MySQL seems to support some form of that by GRANT […] REQUIRE SUBJECT '[…]' (cf.; MariaDB:

While I don't think that our DB setup should be switched any time soon (if that would be possible), it would be nice if the certificates required by this task are future-compatible for such scenarios. So we should test that they are (rudimentarily) usable for MySQL, nginx (proxymanager), etc.

yuvipanda removed yuvipanda as the assignee of this task.Jan 23 2017, 8:57 AM
scfc triaged this task as Low priority.Feb 16 2017, 9:06 PM
scfc moved this task from Triage to Backlog on the Toolforge board.

I chatted with @yuvipanda about this on irc a bit while wondering out loud how to detach the Kubernetes master from NFS which came up while dealing with T169289: Tool Labs 2017-06-29 Labstore100[45] kernel upgrade issues.

A rough plan of the work that would be needed is:

  • Create a signing cert for this use case
  • Split maintain-kubeusers into two parts:
    • One bit generates and signs certs for each tool account. This can run on an NFS master where the certs can be chattr +i to keep them from being tampered with on NFS clients.
    • The other bit generates the ABAC mapping file. This can run locally on each Kubernetes master.

Certificate generation and signing would probably be easiest using openssl cli commands. Its at least theoretically possible to do in Python directly but probably more work than it is worth. There are plenty of tutorials on how to do all the required parts using openssl.

Are there any turnkey solutions for certificate management? Because generating the keypair is the easy part -- signing (and returning the signed cert), renewal and revocation are where it becomes tricky. Having a standard solution could help to manage the complexity.

bd808 edited projects, added Kubernetes; removed Cloud-Services.Jul 28 2017, 11:01 PM
Bstorm added a subscriber: Bstorm.Nov 21 2018, 11:14 PM