Integrate kube-metrics-server into our infrastructure
Open, LowPublic
Actions

Assigned To

None

Authored By

	akosiaris
	Apr 10 2020, 3:45 PM

Description

Per https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/, if we ever want to have an horizontal pod autoscaler, we need to have APIs to feed it data with. The first such API (others will follow) is metrics.k8s.io API, generally provided by metrics-server. It can be launched as a cluster addon or outside of it, as long as it can talk to the kubelets.

The software is at https://github.com/kubernetes-sigs/metrics-server and has a couple of requirements

Metrics server needs to be reachable from kube-apiserver
Kube-apiserver should be correctly configured to enable aggregation layer
Nodes need to have kubelet authorization configured and match metrics-server configuration
Pod/Node metrics need to be exposed by Kubelet by Summary API

For a deployment, we would have to weigh whether we want it in the cluster (and a helm chart already exists for it), or outside the cluster (we would have to write puppet code for it). Both options seem to be supported, we need to investigate.

While this will allow us to more automatically react to changing resource needs, priority for it is relatively low as we don't have that need, our usage levels don't change a lot during the day.

That being said, it's a pretty self contained project and hence patches, ideas, packaging of the metrics-server software would be nice.

Related Objects

Mentioned In: T318925: Getting the Metrics API (K8) functioning to support Auto Scaling
T278192: Install Istio on ml-serve cluster
Mentioned Here: T318925: Getting the Metrics API (K8) functioning to support Auto Scaling

Event Timeline

akosiaris created this task.Apr 10 2020, 3:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 10 2020, 3:45 PM

akosiaris updated the task description. (Show Details)Apr 10 2020, 3:48 PM

I investigated possibility to run metric server outside the cluster. And what I did to run in this way:

Ran k8s cluster
Deployed required manifests APIService from https://github.com/kubernetes-sigs/metrics-server/tree/master/manifests/base (rbac.yaml, apiservice.yaml)
Created Kubectl config

#!/bin/bash
USER_NAME="metrics-server"
K8S_CLUSTER_ENDPOINT="https://192.168.64.4:6443"
CERT_NAME="$USER_NAME.crt"
SECRET_NAME=`kubectl get sa $USER_NAME -n kube-system -o jsonpath='{.secrets[0].name}'`
USER_CA=`kubectl get secret $SECRET_NAME -n kube-system -o jsonpath='{.data.ca\.crt}' | base64 -d > $CERT_NAME`
USER_TOKEN=`kubectl get secret $SECRET_NAME -n kube-system -o jsonpath='{.data.token}' | base64 -d`
KUBECONFIG_NAME="testconfig"


kubectl config set-cluster test-cluster --server=$K8S_CLUSTER_ENDPOINT --certificate-authority=$CERT_NAME --kubeconfig=$KUBECONFIG_NAME
kubectl config set-credentials metrics-server --token=$USER_TOKEN --kubeconfig=$KUBECONFIG_NAME
kubectl config set-context test --cluster test-cluster --user metrics-server --kubeconfig=$KUBECONFIG_NAME
kubectl config use-context test --kubeconfig=$KUBECONFIG_NAME

Ran metrics-server with following args

./metrics-server --authorization-kubeconfig testconfig --cert-dir /tmp --kubeconfig testconfig --authentication-kubeconfig testconfig --kubelet-insecure-tls

Deployed custom endpoint and service without selector in kube-system namespace

apiVersion: v1
kind: Endpoints
metadata:
  name: metrics-server
subsets:
  - addresses:
      - ip: 192.168.64.6
    ports:
      - port: 443

apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: main-port

Test apiservices:

ubuntu@microk8s:~$ kubectl get apiservices v1beta1.metrics.k8s.io
NAME                                   SERVICE                      AVAILABLE   AGE
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        4d22h

Test tops

ubuntu@microk8s:~$ kubectl top nodes
NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
microk8s        301m         7%     2080Mi          54%
microk8s-node   146m         3%     853Mi           22%

I used self-signed certificates and option --kubelet-insecure-tls was required for me and in APIService "insecureSkipVerify: true" as well.

jijiki moved this task from Incoming 🐫 to cross-team active on the serviceops board.Aug 17 2020, 11:45 PM

elukey subscribed.Jun 30 2021, 4:47 PM

elukey mentioned this in T278192: Install Istio on ml-serve cluster.Jul 1 2021, 1:50 PM

BTullis subscribed.Mar 4 2022, 10:52 AM

Joe moved this task from cross-team active to ⎈Kubernetes on the serviceops board.Oct 6 2022, 5:50 AM

JMeybohm triaged this task as Low priority.Oct 10 2022, 7:24 AM

JMeybohm added a project: Kubernetes.

I suggest that we merge T318925: Getting the Metrics API (K8) functioning to support Auto Scaling in as a duplicate of this ticket.

BTullis merged a task: T318925: Getting the Metrics API (K8) functioning to support Auto Scaling .Jul 3 2023, 12:06 PM

BTullis added a subscriber: • EChetty.

JMeybohm added a project: Prod-Kubernetes.Sep 22 2023, 9:01 AM