Page MenuHomePhabricator

Switch to cgroup v2 and systemd as cgroup driver for docker and kubelet
Closed, ResolvedPublic

Description

We're currently running k8s nodes with cgroup v2 disabled (systemd.unified_cgroup_hierarchy=0) as kubelet 1.16 only supports v1. We're also running docker with cgroupfs which is said to cause instability under pressure.

With the k8s 1.23 update we should:

  • Enable cgroup v2: systemd.unified_cgroup_hierarchy=1
  • Docker supports cgroup v2 since Docker 20.10. Running Docker on cgroup v2 also requires the following conditions to be satisfied:
    • containerd: v1.4 or later
    • runc: v1.0.0-rc91 or later
    • Kernel: v4.15 or later (v5.2 or later is recommended)
  • Switch the docker cgroup driver to systemd: native.cgroupdriver=systemd (https://gerrit.wikimedia.org/r/c/operations/puppet/+/524186)
  • Ensure kubelet runs with cgroup driver systemd as well: --cgroup-driver=systemd

https://v1-23.docs.kubernetes.io/docs/setup/production-environment/container-runtimes/#cgroup-drivers
https://v1-23.docs.kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#configuring-a-cgroup-driver

See also: T277876: Reserve resources for system daemons on kubernetes nodes

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptJul 21 2022, 8:34 AM

Change 855567 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Use systemd cgroup (v2) driver with 1.23

https://gerrit.wikimedia.org/r/855567

Change 855567 merged by JMeybohm:

[operations/puppet@production] k8s: Use systemd cgroup (v2) driver with 1.23

https://gerrit.wikimedia.org/r/855567