To do a simple HA setup for Toolforge k8s, we'll need a trio of api servers behind a load balancer. This load balancer is probably best using haproxy rather than nginx (probably?), but we need to check if it needs to be part of tls termination or what have you (hope not, but maybe). This needs puppet stuff and experiments -- and this is what would get the DNS name and the static IP since it should be a fully rebuild-able item vs the api servers which might be harder to fix so easily.
There are other ways to do this (zookeeper and such), but this is the documented and straightforward method. We can also potentially have a passive/active pair of haproxy servers to allow us to move the proxy around if needed fairly simply.
References:
- https://kubernetes.io/docs/setup/independent/high-availability
- https://github.com/kubernetes/kubernetes/issues/18174 (why a LB is needed, instead of kubelet figuring a master is down by itself)