Page MenuHomePhabricator

[infra,k8s] Upgrade Toolforge Kubernetes to version 1.29
Closed, ResolvedPublic

Description

Upgrade procedure: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes

Refer to the link above for the detailed procedure, and update the checkboxes as you complete them.

Before upgrading

  • Check Kubernetes changelog
  • Announce user-facing changes: not required
  • Prepare the new APT packages
  • Upgrade Toolforge components
  • Test new k8s version in lima-kilo

Upgrade toolsbeta cluster

T390212: Upgrade "toolsbeta" cluster to k8s 1.29.15

Upgrade tools cluster

T390214: Upgrade "tools" cluster to k8s 1.29.15

After upgrading

  • Upgrade lima-kilo
  • Upgrade Toolforge components (optional)

Details

Other Assignee
dcaro
Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
builds-api: bump to 0.0.181-20250311143252-180a03bbrepos/cloud/toolforge/toolforge-deploy!711ghostbump_builds-apimain
envvars-admission: bump to 0.0.26-20250311143252-34db248frepos/cloud/toolforge/toolforge-deploy!710ghostbump_envvars-admissionmain
registry-admission: bump to 0.0.58-20250311141826-b329eb93repos/cloud/toolforge/toolforge-deploy!709ghostbump_registry-admissionmain
envvars-api: bump to 0.0.66-20250311141810-1e6c250brepos/cloud/toolforge/toolforge-deploy!708ghostbump_envvars-apimain
ingress-admission: bump to 0.0.58-20250311141819-fc4d75c0repos/cloud/toolforge/toolforge-deploy!707ghostbump_ingress-admissionmain
volume-admission: bump to 0.0.64-20250311141825-ae6fee03repos/cloud/toolforge/toolforge-deploy!706ghostbump_volume-admissionmain
Update deps for K8s 1.29repos/cloud/toolforge/registry-admission!21fnegriT362868main
Update deps for K8s 1.29repos/cloud/toolforge/volume-admission!27fnegriT362868main
Update deps for K8s 1.29repos/cloud/toolforge/ingress-admission!18fnegriT362868main
Update deps for K8s 1.29repos/cloud/toolforge/envvars-api!52fnegriT362868main
Update deps for K8s 1.29repos/cloud/toolforge/envvars-admission!19fnegriT362868main
Update deps for K8s 1.29repos/cloud/toolforge/builds-api!121fnegriT362868main
wmcs-k8s-metrics: upgrade charts for K8s v1.29repos/cloud/toolforge/toolforge-deploy!681fnegriT362868main
Upgrade Kubernetes to 1.29repos/cloud/toolforge/lima-kilo!227fnegrik8s_1.29main
Show related patches Customize query in GitLab

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
fnegri changed the task status from Open to In Progress.Jan 29 2025, 6:42 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-04T14:01:21Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-04T14:01:25Z] <fnegri@cloudcumin1001> Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-04T14:01:33Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-05T17:44:54Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-05T17:44:58Z] <fnegri@cloudcumin1001> Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-05T17:45:08Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T362868)

Change #1126054 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] aptrepo: fetch toolforge k8s v1.29 packages

https://gerrit.wikimedia.org/r/1126054

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/706

volume-admission: bump to 0.0.64-20250311141825-ae6fee03

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/707

ingress-admission: bump to 0.0.58-20250311141819-fc4d75c0

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/708

envvars-api: bump to 0.0.66-20250311141810-1e6c250b

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/709

registry-admission: bump to 0.0.58-20250311141826-b329eb93

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/710

envvars-admission: bump to 0.0.26-20250311143252-34db248f

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/711

builds-api: bump to 0.0.181-20250311143252-180a03bb

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T14:51:45Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T15:01:25Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T362868)

Change #1126054 merged by FNegri:

[operations/puppet@production] aptrepo: fetch toolforge k8s v1.29 packages

https://gerrit.wikimedia.org/r/1126054

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T17:32:27Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T17:32:43Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T17:32:51Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component envvars-api (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T17:33:09Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T17:33:41Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T17:34:18Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T17:35:57Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T17:36:37Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T17:36:47Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component builds-api (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-11T17:46:29Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-12T19:00:55Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T16:14:01Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T16:24:59Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T16:36:30Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T16:46:44Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T17:04:58Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component envvars-api (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T17:14:02Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T17:17:18Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T17:27:45Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T17:29:45Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T17:40:25Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T17:50:09Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component builds-api (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T18:00:56Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T18:04:57Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T362868)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-13T18:14:29Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T362868)

I tested the patch https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/227 in lima-kilo with the following scenarios:

  • all control nodes and worker node running 1.29 (no change from the patch)
  • all control nodes running 1.29 but worker node still on 1.28

I could run all functional tests successfully in both scenarios.

I think we're good to go for upgrading toolsbeta.

fnegri updated the task description. (Show Details)
fnegri updated the task description. (Show Details)
fnegri updated the task description. (Show Details)
fnegri moved this task from In Progress to Done on the Toolforge (Toolforge iteration 19) board.
fnegri moved this task from In progress to Done on the cloud-services-team (FY2024/2025-Q3-Q4) board.