Toolforge k8s: Migrate workers to Containerd and Bookworm
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• taavi
	Jun 9 2021, 2:35 PM

Description

To upgrade Kubernetes 1.24 we need to upgrade Toolforge workers to Containerd. We need Debian 12 for a new enough Containerd version.

toolsbeta

control
worker
ingress

tools

control
worker
ingress

Details

Subject	Repo	Branch	Lines +/-
kubeadm: Drop buster support	operations/puppet	production	+9 -89
toolforge: k8s: Support containerd as container runtime	cloud/wmcs-cookbooks	main	+1 -4
inventory: refresh tools k8s control nodes	cloud/wmcs-cookbooks	main	+1 -1
toolforge: add_k8s_node: Allow passing --network	cloud/wmcs-cookbooks	main	+15 -0
toolforge: add_k8s_node: Add support for containerd	cloud/wmcs-cookbooks	main	+2 -2
P:wmcs::kubeadm: worker: support containerd separate volume	operations/puppet	production	+19 -7
kubeadm: add required config for containerd	operations/puppet	production	+263 -6
kubeadm: containerd: add kernel modules and config	operations/puppet	production	+10 -0
kubeadm: only install containerd.io with docker	operations/puppet	production	+1 -1
P:wmcs::kubeadm: install containerd on bookworm	operations/puppet	production	+23 -10
P:wmcs::kubeadm: rely on iptables-nft on bookworm	operations/puppet	production	+8 -10
aptrepo: Import kubeadm 1.23 for bookworm	operations/puppet	production	+4 -0

Related Objects
Search...

Status	Assigned	Task
Open	None	T362869 [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30)
Open	None	T362868 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29
Open	None	T362867 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28
Open	None	T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
Open	None	T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26
Open	None	T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25
Resolved	aborrero	T307651 Upgrade Toolforge Kubernetes to version 1.24
Open	None	T360699 Toolsbeta: migrate to Debian Bullseye or later
Open	None	T311897 Toolforge: migrate to Debian Bullseye or later
Open	None	T311908 Migrate Toolforge Kubernetes hosts to Debian Bullseye or later
Resolved	• Bstorm	T262550 Toolforge returns HTTP 502 error
Open	• taavi	T262562 [infra] Fix the mis-named k8s service in tools and toolsbeta projects
Resolved	• taavi	T355883 Create a pool of NFS-less Toolforge Kubernetes workers
Resolved	• taavi	T284656 Toolforge k8s: Migrate workers to Containerd and Bookworm
Resolved	• taavi	T349795 Upgrade cadvisor
Resolved	• taavi	T350227 toolforge prometheus servers OOMing
Resolved	• taavi	T357901 Request increased server-group-members quota for tools
Resolved	aborrero	T358476 toolforge k8s: some static pods needs manual restart

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 9 2021, 2:35 PM

• nskaggs triaged this task as Low priority.Aug 10 2021, 10:12 PM

• taavi mentioned this in T280402: Upgrade Toolforge Kubernetes to latest 1.20.Oct 11 2021, 10:51 AM

For this, we currently rely on docker settings to manage log length in containerd, much like prod does. We will want to find an equivalent later because some tools are otherwise very good at filling worker nodes (an old problem around here T148487). logrotate can handle it, but docker was quite good at it with fewer failures waiting for a logrotate run (yes people crashed k8s nodes between logrotate runs regularly, typically using java).

Whatever you use to solve that problem (some containerd setting or podman thing?), just know that our users can certainly outsmart logrotate by mistake.

Looks like https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration has options (not the snazziest when it comes to puppetizing, but you can).

Oh yeah, please don't remove docker-ce from the repos unless you account for the harbor use of it, also. It's running in docker compose and currently using our kubeadm components to do it.

In T284656#7420325, @Bstorm wrote:

Oh yeah, please don't remove docker-ce from the repos unless you account for the harbor use of it, also. It's running in docker compose and currently using our kubeadm components to do it.

I was planning on just using what Debian packages, https://packages.debian.org/bullseye/docker.io and https://apt-browser.toolforge.org/buster-wikimedia/thirdparty/kubeadm-k8s-1-20/ both seem recent enough. Good to know Harbor needs this though, thanks!

Based on https://github.com/containerd/containerd/issues/4830, https://github.com/kubernetes/enhancements/issues/2411, and CRIContainerLogRotation on https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/ I think Kubernetes will automatically deal with log rotation on Containerd as needed.

• taavi added a parent task: T307651: Upgrade Toolforge Kubernetes to version 1.24.May 5 2022, 6:58 AM

• taavi mentioned this in T311908: Migrate Toolforge Kubernetes hosts to Debian Bullseye or later.Jul 2 2022, 12:22 PM

fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:51 PM

fnegri moved this task from Kanban to Inbox on the cloud-services-team board.

Beware: Kubernetes 1.24 requires containerd v1.6.4+ or v1.5.11+, while Bullseye repositories have 1.4.13. Bookworm (bullseye + 1) will ship with 1.6, or we might need to use third-party packages which I'd really rather not do.

• taavi renamed this task from Toolforge k8s: Migrate from Docker to Containerd to Toolforge k8s: Migrate workers to Containerd and Bookworm.Oct 18 2023, 1:04 PM

• taavi raised the priority of this task from Low to Medium.

• taavi updated the task description. (Show Details)

• taavi added a parent task: T311908: Migrate Toolforge Kubernetes hosts to Debian Bullseye or later.

• taavi removed a subscriber: • Bstorm.

Change 967875 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] aptrepo: Import kubeadm 1.23 for bookworm

https://gerrit.wikimedia.org/r/967875

gerritbot added a project: Patch-For-Review.Oct 23 2023, 10:15 AM

Change 968618 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs::kubeadm: rely on iptables-nft on bookworm

https://gerrit.wikimedia.org/r/968618

Change 967875 merged by Majavah:

[operations/puppet@production] aptrepo: Import kubeadm 1.23 for bookworm

https://gerrit.wikimedia.org/r/967875

Mentioned in SAL (#wikimedia-operations) [2023-10-25T10:02:03Z] <taavi> import kubernetes 1.23 packages for debian bookworm T284656

Change 968623 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs::kubeadm: install containerd on bookworm

https://gerrit.wikimedia.org/r/968623

Change 968618 merged by Majavah:

[operations/puppet@production] P:wmcs::kubeadm: rely on iptables-nft on bookworm

https://gerrit.wikimedia.org/r/968618

Change 968623 merged by Majavah:

[operations/puppet@production] P:wmcs::kubeadm: install containerd on bookworm

https://gerrit.wikimedia.org/r/968623

Maintenance_bot removed a project: Patch-For-Review.Oct 25 2023, 10:30 AM

Change 968634 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] kubeadm: only install containerd.io with docker

https://gerrit.wikimedia.org/r/968634

Change 968635 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] kubeadm: containerd: install br_netfilter kmod

https://gerrit.wikimedia.org/r/968635

Change 968647 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] kubeadm: add required config for containerd

https://gerrit.wikimedia.org/r/968647

• taavi edited projects, added Toolforge (Toolforge iteration 01); removed Toolforge.Oct 25 2023, 12:40 PM

• taavi moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 01) board.

• taavi claimed this task.Oct 25 2023, 1:27 PM

dcaro edited projects, added Toolforge (Toolforge iteration 02); removed Toolforge (Toolforge iteration 01).Oct 25 2023, 1:54 PM

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 02) board.

The above patches make it possible to provision a new host on bookworm. There are a couple of issues however:

cadvisor does not work
the extra volume for /var/lib/docker has not been ported yet
I haven't checked if the log file max size still works

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/4

chart: update cadvisor to 0.47.2

• taavi closed subtask T349795: Upgrade cadvisor as Resolved.Oct 30 2023, 2:05 PM

• taavi reopened subtask T349795: Upgrade cadvisor as Open.Nov 1 2023, 2:22 PM

Change 968634 merged by Majavah:

[operations/puppet@production] kubeadm: only install containerd.io with docker

https://gerrit.wikimedia.org/r/968634

Change 968635 merged by Majavah:

[operations/puppet@production] kubeadm: containerd: add kernel modules and config

https://gerrit.wikimedia.org/r/968635

Change 968647 merged by Majavah:

[operations/puppet@production] kubeadm: add required config for containerd

https://gerrit.wikimedia.org/r/968647

dcaro edited projects, added Toolforge (Toolforge iteration 03); removed Toolforge (Toolforge iteration 02).Jan 9 2024, 3:28 PM

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 03) board.

dcaro changed the task status from Open to In Progress.Jan 18 2024, 5:06 PM

• taavi closed subtask T349795: Upgrade cadvisor as Resolved.Jan 24 2024, 9:16 AM

dcaro edited projects, added Toolforge (Toolforge iteration 04); removed Toolforge (Toolforge iteration 03).Jan 24 2024, 9:17 AM

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 04) board.

cadvisor does not work

Fixed with the upgrade.

I haven't checked if the log file max size still works

According to the Kubernetes docs, "containerLogMaxSize is a quantity defining the maximum size of the container log file before it is rotated. For example: "5Mi" or "256Ki". If DynamicKubeletConfig (deprecated; default off) is on, when dynamically updating this field, consider that it may trigger log rotation. Default: "10Mi"". So I think we're fine.

So that leaves the extra volume. It seems like Containerd spreads what Docker stores in /var/lib/docker to a few different places so we need to workaround that somehow.

Change 992633 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs::kubeadm: worker: support containerd separate volume

https://gerrit.wikimedia.org/r/992633

Change 992633 merged by Majavah:

[operations/puppet@production] P:wmcs::kubeadm: worker: support containerd separate volume

https://gerrit.wikimedia.org/r/992633

• taavi added a parent task: T355883: Create a pool of NFS-less Toolforge Kubernetes workers.Jan 25 2024, 1:00 PM

Change 992923 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] toolforge: add_k8s_node: Add support for containerd

https://gerrit.wikimedia.org/r/992923

Change 992926 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] toolforge: add_k8s_node: Allow passing --network

https://gerrit.wikimedia.org/r/992926

• taavi added a parent task: T262562: [infra] Fix the mis-named k8s service in tools and toolsbeta projects.Jan 26 2024, 11:02 AM

Change 992923 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] toolforge: add_k8s_node: Add support for containerd

https://gerrit.wikimedia.org/r/992923

• taavi mentioned this in rCCKBf7b93271fc13: toolforge: add_k8s_node: Add support for containerd.Jan 26 2024, 1:44 PM

Change 992926 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] toolforge: add_k8s_node: Allow passing --network

https://gerrit.wikimedia.org/r/992926

• taavi mentioned this in rCCKBaa0ff4ab1f8b: toolforge: add_k8s_node: Allow passing --network.Jan 29 2024, 6:54 PM

dcaro edited projects, added Toolforge (Toolforge iteration 05); removed Toolforge (Toolforge iteration 04).Feb 6 2024, 5:43 PM

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 05) board.

• taavi updated the task description. (Show Details)Feb 13 2024, 3:18 PM

• taavi mentioned this in T357901: Request increased server-group-members quota for tools.Feb 19 2024, 12:30 PM

• taavi added a subtask: T357901: Request increased server-group-members quota for tools.

• taavi closed subtask T357901: Request increased server-group-members quota for tools as Resolved.Feb 19 2024, 12:34 PM

• taavi updated the task description. (Show Details)Feb 20 2024, 2:17 PM

• taavi updated the task description. (Show Details)Feb 20 2024, 4:07 PM

dcaro edited projects, added Toolforge (Toolforge iteration 06); removed Toolforge (Toolforge iteration 05).Feb 21 2024, 10:10 AM

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 06) board.

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T09:29:38Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster (T284656)

Running cookbook:

aborrero@cloudcumin1001:~ $ sudo cookbook wmcs.toolforge.add_k8s_node --cluster-name tools --task-id T284656 --role control

If this were the first bookworm control node, we would add the argument --image debian-12.0-bookworm, but since it is not, the cookbook will use the last node image.

aborrero added a project: User-aborrero.Feb 22 2024, 9:30 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T11:23:02Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node (T284656)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T11:23:50Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) (T284656)

I plan to upgrade the last control node next monday 2024-02-26.

aborrero moved this task from Backlog to Next on the User-aborrero board.Feb 22 2024, 11:56 AM

Change 1005766 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/wmcs-cookbooks@main] inventory: refresh tools k8s control nodes

https://gerrit.wikimedia.org/r/1005766

Change 1005766 merged by Arturo Borrero Gonzalez:

[cloud/wmcs-cookbooks@main] inventory: refresh tools k8s control nodes

https://gerrit.wikimedia.org/r/1005766

aborrero mentioned this in rCCKB3a57a2190a4d: inventory: refresh tools k8s control nodes.Feb 22 2024, 2:14 PM

Change 1005954 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] toolforge: k8s: Support containerd as container runtime

https://gerrit.wikimedia.org/r/1005954

Change 1005954 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] toolforge: k8s: Support containerd as container runtime

https://gerrit.wikimedia.org/r/1005954

• taavi mentioned this in rCCKB97306ac49c27: toolforge: k8s: Support containerd as container runtime.Feb 23 2024, 10:55 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-26T09:26:11Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster (T284656)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-26T09:53:59Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node (T284656)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-26T09:54:45Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) (T284656)

aborrero mentioned this in rCCKB065ba61a1cfb: inventory: refresh tools-k8s-control nodes.Feb 26 2024, 10:02 AM

This is done:

aborrero@tools-sgebastion-11:~$ kubectl sudo get nodes -o wide | grep control
tools-k8s-control-7       Ready    control-plane,master   5d1h    v1.23.17   172.16.0.144   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-18-cloud-amd64   containerd://1.6.20
tools-k8s-control-8       Ready    control-plane,master   4d1h    v1.23.17   172.16.5.194   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-18-cloud-amd64   containerd://1.6.20
tools-k8s-control-9       Ready    control-plane,master   77m     v1.23.17   172.16.3.135   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-18-cloud-amd64   containerd://1.6.20