Page MenuHomePhabricator

Move kubernetes authentication to using X.509 client certs
Closed, DuplicatePublic

Description

Currently we use tokenauth for authenticating clients to kubernetes servers, which works but requires share state between all the k8s master when running in HA configuration. This 'state' is the tokenauth csv file, and updating this file also requires a k8s apiserver restart, which is not nice.

Instead, we should use X.509 client certs for authentications. We can have a CA setup just for this, and it can be used to sign certs. This way we only need to share the CA between the k8s masters. ABAC info also needs to be shared, but that can be independently generated on each node, I think.

Event Timeline

I had thought about using certificates to authenticate tools in services in the past, mostly so that there would be only one "thing" per tool and not a separate pair of username and password for each service there is.

MySQL seems to support some form of that by GRANT […] REQUIRE SUBJECT '[…]' (cf. https://dev.mysql.com/doc/refman/5.5/en/grant.html; MariaDB: https://mariadb.com/kb/en/mariadb/grant/#per-account-ssltls-options).

While I don't think that our DB setup should be switched any time soon (if that would be possible), it would be nice if the certificates required by this task are future-compatible for such scenarios. So we should test that they are (rudimentarily) usable for MySQL, nginx (proxymanager), etc.

scfc triaged this task as Low priority.Feb 16 2017, 9:06 PM
scfc moved this task from Backlog to Ready to be worked on on the Toolforge board.

I chatted with @yuvipanda about this on irc a bit while wondering out loud how to detach the Kubernetes master from NFS which came up while dealing with T169289: Tool Labs 2017-06-29 Labstore100[45] kernel upgrade issues.

A rough plan of the work that would be needed is:

  • Create a signing cert for this use case
  • Split maintain-kubeusers into two parts:
    • One bit generates and signs certs for each tool account. This can run on an NFS master where the certs can be chattr +i to keep them from being tampered with on NFS clients.
    • The other bit generates the ABAC mapping file. This can run locally on each Kubernetes master.

Certificate generation and signing would probably be easiest using openssl cli commands. Its at least theoretically possible to do in Python directly but probably more work than it is worth. There are plenty of tutorials on how to do all the required parts using openssl.

Are there any turnkey solutions for certificate management? Because generating the keypair is the easy part -- signing (and returning the signed cert), renewal and revocation are where it becomes tricky. Having a standard solution could help to manage the complexity.