Reserve resources for system daemons on kubernetes nodes
Open, MediumPublic
Actions

Assigned To

None

Authored By

	JMeybohm
	Mar 19 2021, 3:37 PM

Description

Currently we allow pods to allocate 100% the resources of a node, which is a bad idea.

We should reserve some CPU, memory maybe storage and PIDs for kubelet (--kube-reserved) as well as for the system itself (--system-reserved). We should also add eviction thresholds.

https://v1-16.docs.kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/

Details

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+2 -0	kubernetes: Switch to using systemd cgroupdriver

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		JMeybohm	T307943 Update Kubernetes clusters to v1.23
		Open		None	T277876 Reserve resources for system daemons on kubernetes nodes

Event Timeline

JMeybohm triaged this task as Medium priority.Mar 19 2021, 3:37 PM

JMeybohm created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 19 2021, 3:37 PM

JMeybohm mentioned this in T278220: Define the size of a pod for mediawiki in terms of resource usage.Mar 23 2021, 11:11 AM

Change 524186 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):
[operations/puppet@production] kubernetes: Switch to using systemd cgroupdriver

https://gerrit.wikimedia.org/r/524186

gerritbot added a project: Patch-For-Review.Mar 26 2021, 8:49 AM

akosiaris added a subscriber: akosiaris.Mar 26 2021, 9:16 AM

JMeybohm mentioned this in T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00).Mar 31 2022, 1:39 PM

JMeybohm mentioned this in T313473: Switch to cgroup v2 and systemd as cgroup driver for docker and kubelet.Jul 21 2022, 8:34 AM

jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:23 PM

jijiki moved this task from 🙈🙉🙊Backlog to ⎈Kubernetes on the serviceops board.Nov 9 2022, 2:14 PM

Change 524186 abandoned by JMeybohm:

[operations/puppet@production] kubernetes: Switch to using systemd cgroupdriver

Reason:

Docker will switch automatically to systemd cgroup driver on cgroupv2 systems

https://gerrit.wikimedia.org/r/524186