production k3s cluster is approaching pod limit
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	SDunlap
	Aug 7 2025, 8:22 PM

Description

Incident

On a recently deploy of patchdemo, the deployment stalled trying to schedule the new pods. We saw the following error:

Warning  FailedScheduling  41m   default-scheduler  0/2 nodes are available: 1 Too many pods. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod..

We were able to resolve this issue by deleting all of the patchdemo-staging environments, and catalyst environments that were more than two weeks old and named test-. (Very sorry if you were using those 😅 )

Configured (default) limits

Our current pod limit for the main k3s node is set to the default recommended limit of 110

kubectl get node k3s -ojsonpath='{.status.capacity.pods}'
110

Implications

When we reach the 110 limit (again), new pods will not be scheduled until some pods are removed. This means that:

patchdemo or catalyst cannot be upgraded/deployed
catalyst will be unable to create new demos
scheduled jobs (like the repo-pool updater and the expiry checker) will not run

Possible remediation

While this limit can be increased, the kubernetes documentation recommends against this, and recommends adding a new node to the cluster instead.

Related Objects

Mentioned Here: T396897: Catalyst configuration as code

Event Timeline

SDunlap created this task.Aug 7 2025, 8:22 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2025, 8:22 PM

SDunlap triaged this task as High priority.Aug 7 2025, 8:22 PM

We can see the number of pods on each node by running:

kubectl get pods -A -o=custom-columns=NODE:.spec.nodeName | sort | uniq -c | sort -n
      1 NODE
      2 k3s-envdb
     84 k3s

SDunlap added subscribers: thcipriani, AMarkossyan-WMF.Aug 7 2025, 8:27 PM

We currently have enough resources to add a new worker node. I was going to wait until completion of T396897 before creating it, but it seems this is the right time to do it

jnuche edited projects, added Catalyst (musi); removed Catalyst.Aug 12 2025, 2:05 PM

jnuche moved this task from Backlog to In progress on the Catalyst (musi) board.

@jnuche and I talked about deploying an node manually (because the opentofu work is still a ways out). Jaime has deployed another node to our horizon project, but it is currently cordoned because we still need to get the repo pool populated on that node (because we are using the local patch storage provisioner. This means the repo-pool jobs will need to run on each node.

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/wiki-repos-pool/-/merge_requests/21

add support for multiple nodes

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/wiki-repos-pool/-/merge_requests/21

add support for multiple nodes

Maintenance_bot removed a project: Patch-For-Review.Aug 13 2025, 12:31 PM

New node k3s-worker01 is now operational.

I could successfully deploy a wiki environment on it:

wiki-189e7ae39e-1676-mediawiki-6d5f947f4c-p5qpt   2/2     Running    0             106s    10.42.2.9     k3s-worker01   <none>           <none>

jnuche moved this task from In progress to Waiting for review on the Catalyst (musi) board.Aug 13 2025, 12:58 PM

jnuche set the point value for this task to 3.

thcipriani added projects: Release-Engineering-Team (Doing 😎), Essential-Work.Aug 15 2025, 11:36 PM

production k3s cluster is approaching pod limitClosed, ResolvedPublic3 Estimated Story PointsActions

Description

Related Objects

Event Timeline

production k3s cluster is approaching pod limit
Closed, ResolvedPublic3 Estimated Story Points
Actions