Per https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/, if we ever want to have an horizontal pod autoscaler, we need to have APIs to feed it data with. The first such API (others will follow) is `metrics.k8s.io` API, generally provided by metrics-server. It can be launched as a cluster addon or outside of it, as long as it can talk to the kubelets.
The software is at https://github.com/kubernetes-sigs/metrics-server and has a couple of requirements
- Metrics server needs to be reachable from kube-apiserver
- Kube-apiserver should be correctly configured to enable aggregation layer
- Nodes need to have kubelet authorization configured and match metrics-server configuration
- Pod/Node metrics need to be exposed by Kubelet by Summary API
For a deployment, we would have to weigh whether we want it in the cluster (and a helm chart already exists for it), or outside the cluster (we would have to write puppet code for it). Both options seem to be supported, we need to investigate.
While this will allow us to more automatically react to changing resource needs, priority for it is relatively low as we don't have that need, our usage levels don't change a lot during the day.
That being said, it's a pretty self contained project and hence patches, ideas, packaging of the metrics-server software would be nice.