Page MenuHomePhabricator
Create Task
Maniphest T390212

Upgrade "toolsbeta" cluster to k8s 1.29.15
Closed, ResolvedPublic
Actions

Description

Upgrade procedure: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes

Refer to the link above for the detailed procedure, and update the checkboxes as you complete them.

If multiple people are working on the upgrade, you can copy the checklist to an Etherpad for easier collaborative editing.

Use this command from a toolforge control node to quickly generate a list of nodes:

for node in $(kubectl get nodes -o json | jq '.items[].metadata.name' -r); do echo "  - [] $node"; done
  • Run functional tests
  • Add a silence in alertmanager
  • Update IRC topic (only for "tools" cluster)
  • Run prepare_upgrade cookbook
  • Upgrade control nodes
    • toolsbeta-test-k8s-control-10
    • toolsbeta-test-k8s-control-11
    • toolsbeta-test-k8s-control-12
  • Upgrade worker nodes
    • toolsbeta-test-k8s-worker-12
    • toolsbeta-test-k8s-worker-13
    • toolsbeta-test-k8s-worker-nfs-10
    • toolsbeta-test-k8s-worker-nfs-5
    • toolsbeta-test-k8s-worker-nfs-7
    • toolsbeta-test-k8s-worker-nfs-8
    • toolsbeta-test-k8s-worker-nfs-9
  • Upgrade ingress nodes
    • toolsbeta-test-k8s-ingress-10
    • toolsbeta-test-k8s-ingress-11
    • toolsbeta-test-k8s-ingress-9
  • Upgrade kubectl on bastions
  • Check everything looks good
  • Remove the silence in alertmanager
  • Revert IRC topic change (only for "tools" cluster)

Related Objects
Search...

Event Timeline

fnegri created this task.Mar 27 2025, 4:32 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMar 27 2025, 4:32 PM
fnegri mentioned this in T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29.Mar 27 2025, 4:35 PM
fnegri updated the task description. (Show Details)Mar 27 2025, 4:51 PM
fnegri changed the task status from Open to In Progress.Mar 31 2025, 11:00 AM
fnegri triaged this task as High priority.
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2024/2025-Q3-Q4) board.
fnegri renamed this task from Upgrade "toolsbeta" cluster to k8s 1.29 to Upgrade "toolsbeta" cluster to k8s 1.29.15.Mar 31 2025, 11:02 AM
Stashbot added a comment.Mar 31 2025, 11:03 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T11:03:57Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 11:04 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T11:04:01Z] <fnegri@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 11:09 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T11:09:37Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 11:10 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T11:10:06Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 (T390212)

fnegri updated the task description. (Show Details)Mar 31 2025, 11:10 AM
Stashbot added a comment.Mar 31 2025, 11:13 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T11:13:46Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 11:42 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T11:42:20Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 (T390212)

fnegri updated the task description. (Show Details)Mar 31 2025, 11:42 AM
Stashbot added a comment.Mar 31 2025, 11:42 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T11:42:56Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 (T390212)

fnegri updated the task description. (Show Details)Mar 31 2025, 11:51 AM
Stashbot added a comment.Mar 31 2025, 11:53 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T11:53:09Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 11:53 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T11:53:25Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 12:08 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T12:08:54Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 (T390212)

fnegri updated the task description. (Show Details)Mar 31 2025, 12:09 PM
fnegri updated the task description. (Show Details)
Stashbot added a comment.Mar 31 2025, 1:02 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:02:54Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:03 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:03:53Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:04 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:04:01Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:05 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:05:11Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:05 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:05:37Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:06 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:06:39Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:07:01Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:07:04Z] <fnegri@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:07:06Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:07:09Z] <fnegri@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:07:58Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:08 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:08:02Z] <fnegri@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:08 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:08:04Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:08 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:08:07Z] <fnegri@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:08 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:08:09Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:08 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:08:12Z] <fnegri@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:10 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:10:14Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:10 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:10:17Z] <fnegri@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:10 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:10:19Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:10 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:10:22Z] <fnegri@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:10 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:10:43Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:10 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:10:46Z] <fnegri@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:11 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:11:23Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:12 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:12:23Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:12 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:12:26Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:13 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:13:24Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:13 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:13:28Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:14 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:14:24Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:14 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:14:27Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:20 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:20:15Z] <fnegri@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:24 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:24:42Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 (T390212)

fnegri updated the task description. (Show Details)Mar 31 2025, 1:25 PM
Stashbot added a comment.Mar 31 2025, 1:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:30:19Z] <fnegri@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:31 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:31:28Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 1:36 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T13:36:43Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 (T390212)

fnegri updated the task description. (Show Details)Mar 31 2025, 1:40 PM
Stashbot added a comment.Mar 31 2025, 2:43 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T14:43:06Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 2:44 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T14:44:01Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 2:45 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T14:45:20Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 2:46 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T14:46:18Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 2:49 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T14:49:01Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 (T390212)

Stashbot added a comment.Mar 31 2025, 2:50 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-31T14:49:59Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 (T390212)

fnegri updated the task description. (Show Details)Mar 31 2025, 2:50 PM
fnegri updated the task description. (Show Details)Mar 31 2025, 4:33 PM
fnegri closed this task as Resolved.Apr 1 2025, 12:48 PM
fnegri updated the task description. (Show Details)
fnegri moved this task from Next Up to Done on the Toolforge (Toolforge iteration 19) board.
fnegri moved this task from In progress to Done on the cloud-services-team (FY2024/2025-Q3-Q4) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits