Page MenuHomePhabricator

Upgrade tools to Kubernetes version 1.33
Closed, ResolvedPublic

Description

Upgrade procedure: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes

Refer to the link above for the detailed procedure, and update the checkboxes as you complete them.

If multiple people are working on the upgrade, you can copy the checklist to an Etherpad for easier collaborative editing.

Use this command from a toolforge control node to quickly generate a list of nodes:

for node in $(kubectl get nodes -o json | jq '.items[].metadata.name' -r); do echo "  - [] $node"; done

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
k8s v1.33.13, helm v3.2.0, kindest/node v1.32.11repos/cloud/toolforge/lima-kilo!338andrewv1.13.13main
Customize query in GitLab

Event Timeline

Andrew renamed this task from Upgrade toolforge to Kubernetes version 1.33 to Upgrade tools to Kubernetes version 1.33.Jul 24 2026, 8:23 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T14:48:45Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T14:49:06Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster tools upgrade to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T14:56:59Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T15:17:04Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T15:19:12Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T15:20:21Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T15:28:04Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132)

upgrade plan for control-7:

[upgrade] Running cluster health checks                                                                                                                                     
[upgrade] Fetching available versions to upgrade to                                                                                                                         
[upgrade/versions] Cluster version: 1.32.13                                                                                                                                 
[upgrade/versions] kubeadm version: v1.33.13                                                                                                                                
[upgrade/versions] Target version: 1.33.13                                                                                                                                  
[upgrade/versions] Latest version in the v1.32 series: 1.33.13                                                                                                              
                                                                                                                                                                            
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':                                                           
COMPONENT   NODE                      CURRENT    TARGET                                                                                                                     
kubelet     tools-k8s-control-7       v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-control-8       v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-control-9       v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-gateway-1       v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-gateway-2       v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-gateway-3       v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-102      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-103      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-105      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-106      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-107      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-108      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-109      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-110      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-111      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-112      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-113      v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-1    v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-10   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-11   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-12   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-13   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-14   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-16   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-17   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-19   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-2    v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-21   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-22   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-23   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-24   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-26   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-27   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-3    v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-32   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-33   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-34   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-35   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-36   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-37   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-38   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-39   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-40   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-41   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-42   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-43   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-44   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-45   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-46   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-47   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-48   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-5    v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-50   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-53   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-54   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-55   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-57   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-58   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-61   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-65   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-66   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-67   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-68   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-69   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-7    v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-70   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-71   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-72   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-73   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-74   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-75   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-76   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-77   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-78   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-79   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-8    v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-80   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-81   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-82   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-83   v1.32.13   1.33.13                                                                                                                    
kubelet     tools-k8s-worker-nfs-9    v1.32.13   1.33.13                                                                                                                    
                                                                                                                                                                            
Upgrade to the latest version in the v1.32 series:                                                                                                                          
                                                                                                                                                                            
COMPONENT                 NODE                  CURRENT    TARGET                                                                                                           
kube-apiserver            tools-k8s-control-7   v1.32.13   1.33.13                                                                                                          
kube-apiserver            tools-k8s-control-8   v1.32.13   1.33.13                                                                                                          
kube-apiserver            tools-k8s-control-9   v1.32.13   1.33.13                                                                                                          
kube-controller-manager   tools-k8s-control-7   v1.32.13   1.33.13                                                                                                          
kube-controller-manager   tools-k8s-control-8   v1.32.13   1.33.13                                                                                                          
kube-controller-manager   tools-k8s-control-9   v1.32.13   1.33.13                                                                                                          
kube-scheduler            tools-k8s-control-7   v1.32.13   1.33.13                                                                                                          
kube-scheduler            tools-k8s-control-8   v1.32.13   1.33.13                                                                                                          
kube-scheduler            tools-k8s-control-9   v1.32.13   1.33.13                                                                                                          
kube-proxy                                      1.32.13    1.33.13                                                                                                          
CoreDNS                                         v1.11.3    v1.12.0                                                                                                          
                                                                                                                                                                            
You can now apply the upgrade by executing the following command:                                                                                                           
                                                                                                                                                                            
        kubeadm upgrade apply 1.33.13                                                                                                                                       
                                                                                                                                                                            
_____________________________________________________________________                                                                                                       
                                                                                                                                                                            
                                                                                                                                                                            
The table below shows the current state of component configs as understood by this version of kubeadm.                                                                      
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or                                                                     
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually                                                                         
upgrade to is denoted in the "PREFERRED VERSION" column.                                                                                                                    
                                                                                                                                                                            
API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED                                                                                     
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no                                                                                                          
kubelet.config.k8s.io     v1beta1           v1beta1             no                                                                                                          
_____________________________________________________________________

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T15:30:40Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T15:31:42Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T15:38:09Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T15:42:09Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.restart_static_pods for tools-k8s-control-7 (T433132)

After upgrading tools-k8s-control-7, the api server yaml file was out of place so the server couldn't restart. Taavi replaced the file and I restarted things with

sudo cookbook wmcs.toolforge.k8s.restart_static_pods --cluster-name tools --task-id T433132 --hostname tools-k8s-control-7

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T15:48:45Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for tools-k8s-control-7 (T433132)

Change #1329597 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] k8s: Do not manually restart kube-apiserver

https://gerrit.wikimedia.org/r/1329597

Change #1329597 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] k8s: Do not manually restart kube-apiserver

https://gerrit.wikimedia.org/r/1329597

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:19:46Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:24:32Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:24:59Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:30:56Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:33:04Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:43:55Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:44:50Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:46:14Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:48:40Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:49:39Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:54:24Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T16:55:17Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T17:00:09Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud (T433132)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-26T17:00:31Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud (T433132)

Andrew updated the task description. (Show Details)