Page MenuHomePhabricator
Create Task
Maniphest T427919

Upgrade tools cluster to Kubernetes 1.32
Closed, ResolvedPublic
Actions

Description

Upgrade procedure: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes

Refer to the link above for the detailed procedure, and update the checkboxes as you complete them.

If multiple people are working on the upgrade, you can copy the checklist to an Etherpad for easier collaborative editing.

Related Objects
Search...

Event Timeline

taavi created this task.Jun 2 2026, 12:58 PM
taavi triaged this task as High priority.Jun 2 2026, 1:59 PM
Stashbot added a comment.Jun 15 2026, 11:00 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:00:36Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:20 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:20:00Z] <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:20 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:20:44Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.31.14 to 1.32.13 (T427919)

taavi added a comment.Jun 15 2026, 11:22 AM
----- OUTPUT for command #1: 'sudo -i kubeadm ...ade plan 1.32.13' -----                                                                                                                                                                 
[preflight] Running pre-flight checks.                                                                                                                                                                                                   
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...                                                                                                                                 
[upgrade/config] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.                                                                                                                                       
[upgrade] Running cluster health checks                                                                                                                                                                                                  
[upgrade] Fetching available versions to upgrade to                                                                                                                                                                                      
[upgrade/versions] Cluster version: 1.31.14                                                                                                                                                                                              
[upgrade/versions] kubeadm version: v1.32.13                                                                                                                                                                                             
[upgrade/versions] Target version: 1.32.13                                                                                                                                                                                               
[upgrade/versions] Latest version in the v1.31 series: 1.32.13                                                                                                                                                                           
                                                                                                                                                                                                                                         
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':                                                                                                                        
COMPONENT   NODE                      CURRENT    TARGET                                                                                                                                                                                  
kubelet     tools-k8s-control-7       v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-control-8       v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-control-9       v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-gateway-1       v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-gateway-2       v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-gateway-3       v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-102      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-103      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-105      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-106      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-107      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-108      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-109      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-110      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-111      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-112      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-113      v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-1    v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-10   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-11   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-12   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-13   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-14   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-16   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-17   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-19   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-2    v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-21   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-22   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-23   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-24   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-26   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-27   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-3    v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-32   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-33   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-34   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-35   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-36   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-37   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-38   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-39   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-40   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-41   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-42   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-43   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-44   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-45   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-46   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-47   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-48   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-5    v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-50   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-53   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-54   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-55   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-57   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-58   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-61   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-65   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-66   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-67   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-68   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-69   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-7    v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-70   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-71   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-72   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-73   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-74   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-75   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-76   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-77   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-78   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-79   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-8    v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-80   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-81   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-82   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-83   v1.31.14   1.32.13                                                                                                                                                                                 
kubelet     tools-k8s-worker-nfs-9    v1.31.14   1.32.13                                                                                                                                                                                 
                                                                                                                                                                                                                                         
Upgrade to the latest version in the v1.31 series:                                                                                                                                                                                       
                                                                                                                                                                                                                                         
COMPONENT                 NODE                  CURRENT    TARGET                                                                                                                                                                        
kube-apiserver            tools-k8s-control-7   v1.31.14   1.32.13                                                                                                                                                                       
kube-apiserver            tools-k8s-control-8   v1.31.14   1.32.13                                                                                                                                                                       
kube-apiserver            tools-k8s-control-9   v1.31.14   1.32.13                                                                                                                                                                       
kube-controller-manager   tools-k8s-control-7   v1.31.14   1.32.13                                                                                                                                                                       
kube-controller-manager   tools-k8s-control-8   v1.31.14   1.32.13                                                                                                                                                                       
kube-controller-manager   tools-k8s-control-9   v1.31.14   1.32.13                                                                                                                                                                       
kube-scheduler            tools-k8s-control-7   v1.31.14   1.32.13                                                                                                                                                                       
kube-scheduler            tools-k8s-control-8   v1.31.14   1.32.13                                                                                                                                                                       
kube-scheduler            tools-k8s-control-9   v1.31.14   1.32.13                                                                                                                                                                       
kube-proxy                                      1.31.14    1.32.13                                                                                                                                                                       
CoreDNS                                         v1.11.3    v1.11.3                                                                                                                                                                       
                                                                                                                                                                                                                                         
You can now apply the upgrade by executing the following command:                                                                                                                                                                        
                                                                                                                                                                                                                                         
        kubeadm upgrade apply 1.32.13                                                                                                                                                                                                    
                                                                                                                                                                                                                                         
_____________________________________________________________________                                                                                                                                                                    
                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                         
The table below shows the current state of component configs as understood by this version of kubeadm.                                                                                                                                   
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or                                                                                                                                  
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually                                                                                                                                      
upgrade to is denoted in the "PREFERRED VERSION" column.                                                                                                                                                                                 
                                                                                                                                                                                                                                         
API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED                                                                                                                                                  
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no                                                                                                                                                                       
kubelet.config.k8s.io     v1beta1           v1beta1             no                                                                                                                                                                       
_____________________________________________________________________                                                                                                                                                                    
                                                                                                                                                                                                                                         
================
Stashbot added a comment.Jun 15 2026, 11:27 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:27:47Z] <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:27 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:27:55Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:30 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:30:21Z] <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-8 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:33 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:33:18Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:39 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:39:01Z] <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:40 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:40:14Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 (T427919)

Stashbot added a comment.Jun 15 2026, 11:53 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:53:01Z] <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 (T427919)

Stashbot added a comment.Jun 15 2026, 11:54 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:54:40Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:55 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:55:34Z] <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:56 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:56:24Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:57 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:57:17Z] <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:58 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:58:15Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 11:59 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T11:59:09Z] <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 (T427919)

Stashbot added a comment.Jun 15 2026, 12:01 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T12:01:30Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud (T427919)

Stashbot added a comment.Jun 15 2026, 12:02 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-15T12:02:08Z] <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud (T427919)

taavi updated the task description. (Show Details)Jun 15 2026, 12:03 PM
taavi closed this task as Resolved.Jun 15 2026, 12:44 PM
taavi updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits