Currently we use tokenauth for authenticating clients to kubernetes servers, which works but requires share state between all the k8s master when running in HA configuration. This 'state' is the tokenauth csv file, and updating this file also requires a k8s apiserver restart, which is not nice.
Instead, we should use X.509 client certs for authentications. We can have a CA setup just for this, and it can be used to sign certs. This way we only need to share the CA between the k8s masters. ABAC info also needs to be shared, but that can be independently generated on each node, I think.