Page MenuHomePhabricator

[k8s,infra] consider scaling the k8s control plane
Closed, ResolvedPublic

Description

As of this writing, some of the k8s control plane components, in particular the k8s apiserver, eats a good chunk of the resources of the virtual machine they run on.

image.png (859×239 px, 60 KB)

For example, from the k9s screenshot above, the apiserver is using:

  • 7.2GB of RAM on control-7
  • 3.6GB of RAM on control-8
  • 4.5GB of RAM on control-9

As of this writing, the VM have 15GB RAM total, which may give little room for the system to survive load spikes and other unexpected scenarios.

This ticket is to evaluate if we should scale the k8s control plane, either or both:

  • horizontally (more nodes)
  • vertically (bigger nodes)

Event Timeline

aborrero changed the task status from Open to In Progress.
aborrero triaged this task as High priority.
aborrero moved this task from Backlog to Next on the User-aborrero board.
dcaro renamed this task from toolforge: consider scaling the k8s control plane to [k8s,infra] consider scaling the k8s control plane.Jun 13 2024, 9:47 AM

I would like to resize the VMs to have 64GB RAM, given we have evidences of the apiserver alone consuming about 8GB.

Having this memory head room can help us "absorb" some memory-intense scenarios which would otherwise result in an outage.

Also, per my tests for T367386: [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways increasing the memory available for the k8s api-server is key for our kyverno setup.

On IRC, @taavi suggested to do 32GB for now. I'm fine with that too.

We will resize like this:

  • from g3.cores8.ram16.disk20
  • to g3.cores8.ram32.disk20