Page MenuHomePhabricator

Integrate kube-metrics-server into our infrastructure
Open, LowPublic

Description

Per https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/, if we ever want to have an horizontal pod autoscaler, we need to have APIs to feed it data with. The first such API (others will follow) is metrics.k8s.io API, generally provided by metrics-server. It can be launched as a cluster addon or outside of it, as long as it can talk to the kubelets.

The software is at https://github.com/kubernetes-sigs/metrics-server and has a couple of requirements

  • Metrics server needs to be reachable from kube-apiserver
  • Kube-apiserver should be correctly configured to enable aggregation layer
  • Nodes need to have kubelet authorization configured and match metrics-server configuration
  • Pod/Node metrics need to be exposed by Kubelet by Summary API

For a deployment, we would have to weigh whether we want it in the cluster (and a helm chart already exists for it), or outside the cluster (we would have to write puppet code for it). Both options seem to be supported, we need to investigate.

While this will allow us to more automatically react to changing resource needs, priority for it is relatively low as we don't have that need, our usage levels don't change a lot during the day.

That being said, it's a pretty self contained project and hence patches, ideas, packaging of the metrics-server software would be nice.

Event Timeline

I investigated possibility to run metric server outside the cluster. And what I did to run in this way:

  1. Ran k8s cluster
  2. Deployed required manifests APIService from https://github.com/kubernetes-sigs/metrics-server/tree/master/manifests/base (rbac.yaml, apiservice.yaml)
  3. Created Kubectl config
#!/bin/bash
USER_NAME="metrics-server"
K8S_CLUSTER_ENDPOINT="https://192.168.64.4:6443"
CERT_NAME="$USER_NAME.crt"
SECRET_NAME=`kubectl get sa $USER_NAME -n kube-system -o jsonpath='{.secrets[0].name}'`
USER_CA=`kubectl get secret $SECRET_NAME -n kube-system -o jsonpath='{.data.ca\.crt}' | base64 -d > $CERT_NAME`
USER_TOKEN=`kubectl get secret $SECRET_NAME -n kube-system -o jsonpath='{.data.token}' | base64 -d`
KUBECONFIG_NAME="testconfig"


kubectl config set-cluster test-cluster --server=$K8S_CLUSTER_ENDPOINT --certificate-authority=$CERT_NAME --kubeconfig=$KUBECONFIG_NAME
kubectl config set-credentials metrics-server --token=$USER_TOKEN --kubeconfig=$KUBECONFIG_NAME
kubectl config set-context test --cluster test-cluster --user metrics-server --kubeconfig=$KUBECONFIG_NAME
kubectl config use-context test --kubeconfig=$KUBECONFIG_NAME
  1. Ran metrics-server with following args
./metrics-server --authorization-kubeconfig testconfig --cert-dir /tmp --kubeconfig testconfig --authentication-kubeconfig testconfig --kubelet-insecure-tls
  1. Deployed custom endpoint and service without selector in kube-system namespace
apiVersion: v1
kind: Endpoints
metadata:
  name: metrics-server
subsets:
  - addresses:
      - ip: 192.168.64.6
    ports:
      - port: 443
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: main-port
  1. Test apiservices:
ubuntu@microk8s:~$ kubectl get apiservices v1beta1.metrics.k8s.io
NAME                                   SERVICE                      AVAILABLE   AGE
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        4d22h
  1. Test tops
ubuntu@microk8s:~$ kubectl top nodes
NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
microk8s        301m         7%     2080Mi          54%
microk8s-node   146m         3%     853Mi           22%

I used self-signed certificates and option --kubelet-insecure-tls was required for me and in APIService "insecureSkipVerify: true" as well.

JMeybohm added a project: Kubernetes.