Umbrella task to track the work required towards upgrading our Kubernetes clusters to Kubernetes 1.34 (EOL: 2026-10-27)

We're currently running 1.31 which went EOL on 2026-02-28.

I've put together some general steps on how we planned/executed kubernetes updates in the past at https://wikitech.wikimedia.org/wiki/Kubernetes/Kubernetes_Infrastructure_Upgrade_Procedure

New update procedure

With this upgrade we're planning to change the way we do upgrades from re-initializing all clusters to rolling updates according to the Version Skew Policy. Since we have not done this yet, we should define the process first. I think we have two options, assuming we want to go from 1.31 to 1.36:

Conservative approach
Upgrade just one minor version to stay within the supported version skew for k8s control-plane components (apiserver, controller-manager, scheduler):

• Update one control-plane, observe changed behavior, resource usage etc. to 1.32
• If everything seems well, update all other control-planes to 1.32
• Update workers in small to big batches to 1.32
• Repeat until we're at 1.36

Pro:

• Playing safe, multiple stages where we can stop and roll back or otherwise intervene
• No control-plane downtime

Con;

• Slow/more work since we need to package and upgrade to every minor version

Aggressive approach
Upgrade all control-planes at once, staying within the supported version skew for k8s worker components (kubelet, kube-proxy):

• Upgrade all control-planes at once to 1.34
• Update workers in small to big batches to 1.34
• Repeat until we're at 1.36

Pro:

• Faster: We may skip up to two k8s versions

Con:

• Some risk involved upgrading all control-planes at the same time
  • Technically we can roll back as long as no incompatible API changes have been made to etcd (by users, kubernetes does not do that on it's own).
• We might need a small window of control-plane downtime (not sure, might just work)

Middleground approach
We could combine both of the above:

• Upgrade control-planes using the conservative approach to 1.32, then 1.33, then 1.34
• Upgrade workers in small to big batches to 1.34
• Repeat until we're at 1.36

Pro:

• Playing safe, multiple stages where we can stop and roll back or otherwise intervene
• No control-plane downtime

Con:

• Slow/more work since we need to package and upgrade control-planes to every minor version

Together with the Kubernetes version we might need to update some components.

Core components (more or less on all clusters)

• calico (currently 3.29.1, target 3.30), compatibility matrix
• istio (currently 1.24.2, target 1.29) compatibility matrix
• cert-manager (currently 1.16.3, target 1.19) compatibility matrix
• coredns (currently 1.11.3, target 1.12.1) version installed by kubeadm
• helm (currently 3.11.3, target 3.19) compatibility matrix
• kube-state-metrics (currently 2.14, target 2.18) compatibility matrix

Operators/Addons (only on specific clusters)

• ✅ kserve (currently 0.17) compatibility matrix
  • 0.17 requires k8s >=1.32, cert-manager >=1.15.0
  • This now lives in knative-resrouces and knative-crd-minimal (T426823)
• ✅ knative-serving (currently 1.21.2) compatibility matrix
  • 1.21 requires k8s >= 1.33
• ✅ apache flink-kubernetes-operator (currently 1.14)
  • Requires k8s >= 1.16
• ✅ spark-operator (currently 2.2.1)compatibility matrix
  • Requires k8s >= 1.16
  • n.b. This is still not yet in active production use for spark workloads, at present.
• ⚠️ cloudnative-pg operator (currently 1.27) compatibility matrix
  • 1.27 supports k8s >=1.31,<=1.33
  • 1.29 does not support 1.31 but it's listed as tested version
  • T424493: Upgrade the cloudnative-pg operator to a version supporting Kubernetes 1.31
• ⚠️ ceph-csi plugins (currently 3.7.2) compatibility matrix
  • 3.7.2 supports k8s >=1.22,<=1.24
  • T407166: Upgrade the ceph-csi-plugin to the latest release compatible with kubernetes version 1.31
• ⚠️ opensearch-operator (currently 2.7)
  • No clear kubernetes compatibility matrix
  • T414687: Deploy the opensearch operator 2.8.0 to dse-k8s-codfw
  • T414217: ☂️Test OpenSearch k8s Operator 3.x ☂️
• ✅ TopoLVM (currently 0.38.1) compatibility notes
  • This automatically tracks the latest three releases of Kubernetes
  • 0.41.0 supports k8s >= 1.33, <=1.35
  • 0.42.0 will likely support k8s >= 1.34, <=1.36
  • 0.43.0 will likely drop support for k8s <= 1.34

Preparation for the Kubernetes update

• Define target Kubernetes version
• Ensure all our charts are compatible with the new Kubernetes version
• Read Kubernetes changelogs (yellow/red flags just linked below each version. Tick the box if all action required items have been addressed, use ✅ for single items)

Changelogs can be found at:

• Per minor changelog file. e.g.: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.31.md
• https://relnotes.k8s.io/ filter/search release notes and changelogs per version and/or component. Helps best when you're looking for something specific.

Template per k8s minor version:

[] [[ https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-9.9.md | v9.9.99 ]]
* **Action Required**
  * **Metrics changes**
* **Node**

Upgrade process

• Build updated Kubernetes packages
• Build updates component packages/container images and update helm charts
• Update staging
• Update grafana dashboards and alerts (to find dashboards using a specific metric, see https://wikitech.wikimedia.org/wiki/Grafana#Search/audit_metrics_usage_across_dashboards)
• Update production