Page MenuHomePhabricator

Figure out cert management for Toolforge kubernetes and make it clear in documents, etc. for the upgrade
Closed, ResolvedPublic


Right now, most k8s services, including etcd use puppet certs (which are valid x.509 certs that work with the puppetmaster itself as the CA). The problem with this is that puppet is not entirely designed to be a CA for other services so they cryptically suggest you not introduce SANs to certificates for clients (only for the puppetmaster).

As is evident in the subtasks of this, SANs (Alternate Names) are needed to make k8s consistent between DNS and the certs actually used.

It is also worth noting that client certs don't need to be issued by the same CA as long as the configs specified to the various services know which cacert to validate them against. This suggests that we could use puppet for client certs even if we chose to not using it for the server side certs (at the cost of some complexity).

Using puppet is somewhat attractive because we already have that infrastructure in place, and it distributes certs for us (or at least requires us to deal with its quirks already). Puppet certs do not help with user certs--x.509 certs can be distributed for user groups, and that seems like the kind of thing maintain-kubeusers would be great for with lots of changes. Turnkey solutions for PKI on Linux are interesting, but it may be just as well to use a script and maintain a "CA" on one instance or another. The kubernetes api-server can be a CA, but we are aiming for HA there.

There's enough here to initially gather tasks under and then merge them as we figure out what we are changing/fixing/solving/rejecting.

Related Objects

Event Timeline

Bstorm renamed this task from Figure out cert management for kubernetes and make it clear in documents, etc. for the upgrade to Figure out cert management for Toolforge kubernetes and make it clear in documents, etc. for the upgrade.Feb 7 2019, 9:03 PM
Bstorm triaged this task as High priority.
Bstorm created this task.
aborrero added a subscriber: aborrero.

I will give this a try soon.

I've been testing generating puppet certs with SANs with no issues so far. Is just a hiera override and re-generate the certs. Steps, for the record:

  1. puppet agent running clean in the affected VM
  2. hiera config (horizon)
profile::base::puppet::dns_alt_names: myAltName1,MyAltName2
  1. run puppet agent to see change in /etc/puppet/puppet.conf, you will se something like:
server = openstack-puppetmaster-01.openstack.eqiad.wmflabs
dns_alt_names = myAltName1,myAltName2
  1. drop SSL config in the puppet client, the usual rm -rf /var/lib/puppet/ssl
  2. cleanup old certificates in the puppemaster: sudo puppet cert clean $FQDN
  3. ask again for the client certs, in the VM: sudo puppet agent -tv
  4. accept the certificate in the master, note we allow now the SANs: sudo puppet cert --allow-dns-alt-names sign $FQDN
  5. run puppet agent again in the VM: sudo puppet agent -tv

I even wrote a quick & dirty cumin-based script to automate most of the dance, and you can test it in labpuppetmaster (the cumin server) and my testing VMs in the openstack CloudVPS project.

The result is:

root@arturo-k8s-test-2:~# openssl x509 -in /var/lib/puppet/ssl/certs/arturo-k8s-test-2.openstack.eqiad.wmflabs.pem -text -noout
            X509v3 Subject Alternative Name: 
                DNS:arturo-k8s-test-2.openstack.eqiad.wmflabs, DNS:myAltName1, DNS:myAltName2

Looking at the required certificates for K8s ( it seems we already have most of them covered by means of puppet?
The only weird thing I see is that system:masters requirement for the O= in the Subject.

So I had this question: How is the prod k8s doing? modules/k8s/manifests/ssl.pp seems to suggest that it just uses puppet certs, confirmed by cumin:

aborrero@cumin1001:~ $ sudo cumin --force "P{R:Class = k8s::apiserver}" 
5 hosts will be targeted:

This is all for the basic k8s infra. I've not started yet investigating user certificates, and that seems to be a different challenge in the case of Toolforge.

Since we now have a proposal for this, let's write it up in

I think what we have now in toolsbeta is going to be great!

@Bstorm I'm assigning this task to you since its mostly you working on this right now (the maintain-kubeusers).

certdesign4k8s.png (399×733 px, 56 KB)

This describes essentially what we are now doing. Etcd client and server certs are simply the puppet certs (which should keep etcd flexible in case we need to set up routing into calico somewhere), while certs for users are x.509s generated using the certificates API of k8s. Node certs are generated by k8s as well using kubeadm (which interacts with the certs API using tokens). The certs to manage the CA and PKI are copied between k8s control plane nodes at build time. A new cluster will have a new CA, which honestly prevents leakage nicely.

The admin users mentioned in that diagram are still theoretical, but there is no reason to require root to interact with a k8s API. It should be straightforward to add a service or a manually run script that maps the <project>.admin group to admin user accounts and places them in the appropriate locations. That will allow Toolforge admins to interact with k8s as easily as they can Grid Engine (and nobody else--they need to use tool accounts). This should simplify playbooks and procedures for dealing with jobs and services that are misbehaving, etc.

Adding to discussion in order to discuss the proposal for admin users since that is a change from the behavior of the original system as well as to open the design proposal for comment/questions/rejection/redo in general.

Another proposal is enabling automatic rotation for kubelet certs so we don't have to manually re-issue them if we don't upgrade during the course of a year. Since upgrading via kubeadm does rotate the certs for all nodes, as long as there is at least one upgrade during a year, we'll be ok, but why chance it?

It will require reconfiguring kubelets and restarting them. It's best if this is included in the kubeadm init config (though it might actually be necessary to do it via puppet or something intstead since some settings don't propagate on join, if I recall.)

Ideas from 2019-08-27 team meeting:

  • Make sure we have monitoring and alerting for cert expiration
  • stick with default 1 year or bump up to 2 year default
  • plan on 6 month upgrade cycle for Kubernetes itself

As of k8s 1.8, I think there's a prometheus metric for cert expiry

We'll get to see what that looks like when a more recent kubelet has prometheus monitoring.

Bstorm changed the task status from Open to Stalled.Nov 8 2019, 11:34 PM

So this is on hold waiting for monitoring to show the new kubelets (which it should soon if it doesn't already) T237643

Change 550673 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: new k8s: add script

Change 550673 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: new k8s: add script

@aborrero Do you feel we are done with this at this point? We may have more to do like recovery procedures, but it feels pretty good for now.

The docs are more or less in shape. We may want to create a section about how to re-encrypt (what almost happened the other day when we lost labs/private), but other than that, yeah.