I've been testing generating puppet certs with SANs with no issues so far. Is just a hiera override and re-generate the certs. Steps, for the record:

1. puppet agent running clean in the affected VM
2. hiera config (horizon)

profile::base::puppet::dns_alt_names: myAltName1,MyAltName2

3. run puppet agent to see change in /etc/puppet/puppet.conf, you will se something like:

[agent]
server = openstack-puppetmaster-01.openstack.eqiad.wmflabs
dns_alt_names = myAltName1,myAltName2

4. drop SSL config in the puppet client, the usual rm -rf /var/lib/puppet/ssl
5. cleanup old certificates in the puppemaster: sudo puppet cert clean $FQDN
6. ask again for the client certs, in the VM: sudo puppet agent -tv
7. accept the certificate in the master, note we allow now the SANs: sudo puppet cert --allow-dns-alt-names sign $FQDN
8. run puppet agent again in the VM: sudo puppet agent -tv

I even wrote a quick & dirty cumin-based script to automate most of the dance, and you can test it in labpuppetmaster (the cumin server) and my testing VMs in the openstack CloudVPS project.
https://wikitech.wikimedia.org/wiki/User:Arturo_Borrero_Gonzalez#switch_puppetmaster_for_CloudVPS_VMs

The result is:

root@arturo-k8s-test-2:~# openssl x509 -in /var/lib/puppet/ssl/certs/arturo-k8s-test-2.openstack.eqiad.wmflabs.pem -text -noout
Certificate:
[..]
            X509v3 Subject Alternative Name: 
                DNS:arturo-k8s-test-2.openstack.eqiad.wmflabs, DNS:myAltName1, DNS:myAltName2
[..]

Looking at the required certificates for K8s (https://kubernetes.io/docs/setup/certificates/#all-certificates) it seems we already have most of them covered by means of puppet?
The only weird thing I see is that system:masters requirement for the O= in the Subject.

So I had this question: How is the prod k8s doing? modules/k8s/manifests/ssl.pp seems to suggest that it just uses puppet certs, confirmed by cumin:

aborrero@cumin1001:~ $ sudo cumin --force "P{R:Class = k8s::apiserver}" 
5 hosts will be targeted:
acrab.codfw.wmnet,acrux.codfw.wmnet,argon.eqiad.wmnet,chlorine.eqiad.wmnet,neon.eqiad.wmnet

This is all for the basic k8s infra. I've not started yet investigating user certificates, and that seems to be a different challenge in the case of Toolforge.

Since we now have a proposal for this, let's write it up in https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals

I think what we have now in toolsbeta is going to be great!

This describes essentially what we are now doing. Etcd client and server certs are simply the puppet certs (which should keep etcd flexible in case we need to set up routing into calico somewhere), while certs for users are x.509s generated using the certificates API of k8s. Node certs are generated by k8s as well using kubeadm (which interacts with the certs API using tokens). The certs to manage the CA and PKI are copied between k8s control plane nodes at build time. A new cluster will have a new CA, which honestly prevents leakage nicely.

The admin users mentioned in that diagram are still theoretical, but there is no reason to require root to interact with a k8s API. It should be straightforward to add a service or a manually run script that maps the <project>.admin group to admin user accounts and places them in the appropriate locations. That will allow Toolforge admins to interact with k8s as easily as they can Grid Engine (and nobody else--they need to use tool accounts). This should simplify playbooks and procedures for dealing with jobs and services that are misbehaving, etc.

Adding to discussion in order to discuss the proposal for admin users since that is a change from the behavior of the original system as well as to open the design proposal for comment/questions/rejection/redo in general.

Another proposal is enabling automatic rotation for kubelet certs so we don't have to manually re-issue them if we don't upgrade during the course of a year. Since upgrading via kubeadm does rotate the certs for all nodes, as long as there is at least one upgrade during a year, we'll be ok, but why chance it? https://kubernetes.io/docs/tasks/tls/certificate-rotation/#enabling-client-certificate-rotation

It will require reconfiguring kubelets and restarting them. It's best if this is included in the kubeadm init config (though it might actually be necessary to do it via puppet or something intstead since some settings don't propagate on join, if I recall.)

Ideas from 2019-08-27 team meeting:

• Make sure we have monitoring and alerting for cert expiration
• stick with default 1 year or bump up to 2 year default
• plan on 6 month upgrade cycle for Kubernetes itself

As of k8s 1.8, I think there's a prometheus metric for cert expiry https://github.com/kubernetes/kubernetes/pull/51031

We'll get to see what that looks like when a more recent kubelet has prometheus monitoring.

Change 550673 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: new k8s: add wmcs-k8s-get-cert.sh script

https://gerrit.wikimedia.org/r/550673

Change 550673 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: new k8s: add wmcs-k8s-get-cert.sh script

https://gerrit.wikimedia.org/r/550673

I created this wikitech page: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Certificates_in_k8s

@aborrero Do you feel we are done with this at this point? We may have more to do like recovery procedures, but it feels pretty good for now.