Page MenuHomePhabricator
Create Task
Maniphest T362867

[infra,k8s] Upgrade Toolforge Kubernetes to version 1.28
Closed, ResolvedPublic
Actions

Description

K8s

https://v1-28.docs.kubernetes.io/blog/2023/08/15/kubernetes-v1-28-release/

Working etherpad: https://etherpad.wikimedia.org/p/k8s-1.27-to-1.28-upgrade
Persistent wiki page: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes/1.27_to_1.28_notes

Workgroup page: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Ongoing_Efforts/Toolforge_Upgrade_Workgroup/Upgrades_Overview

Components

Pre-k8s upgrade

can be upgraded (potentially not blocking, tests pass without them upgrading)

Post-k8s upgrade

need upgrading
can be upgraded
  • volume-admission, k8s.io deps to 0.28.X
  • registry-admission, k8s.io deps to 0.28.X
  • ingress-admission, k8s.io deps to 0.28.X
  • envvars-api, k8s.io deps to 0.28.X

Details

Other Assignee
fnegri
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
aptrepo: add k8s 1.28 reposoperations/puppetproduction+11 -11
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
volume-admission: bump to 0.0.59-20241120000415-71c39564repos/cloud/toolforge/toolforge-deploy!610ghostbump_volume-admissionmain
[volume-admission] k8s.io deps to 0.28.14repos/cloud/toolforge/volume-admission!21raymond-ndibek8sio_dep_updatemain
envvars-api: bump to 0.0.62-20241119190711-285eda5crepos/cloud/toolforge/toolforge-deploy!609ghostbump_envvars-apimain
registry-admission: bump to 0.0.54-20241119190632-ec34b7a8repos/cloud/toolforge/toolforge-deploy!608ghostbump_registry-admissionmain
ingress-admission: bump to 0.0.54-20241119090358-300b9ae5repos/cloud/toolforge/toolforge-deploy!602ghostbump_ingress-admissionmain
[ingress-admission] k8s.io deps to 0.28.14repos/cloud/toolforge/ingress-admission!14raymond-ndibek8sio_dep_updatemain
[registry-admission] k8s.io deps to 0.28.14repos/cloud/toolforge/registry-admission!17raymond-ndibek8sio_dep_updatemain
[envvars-api] k8s.io deps to 0.28.14repos/cloud/toolforge/envvars-api!48raymond-ndibek8sio_dep_updatemain
[toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0repos/cloud/toolforge/toolforge-deploy!586raymond-ndibeupdate_kube_state_metricsmain
[toolforge-deploy] update calico kubeVersionrepos/cloud/toolforge/toolforge-deploy!585raymond-ndibefix_calico_kube_versionmain
[lima-kilo] test k8s 1.28 upgraderepos/cloud/toolforge/lima-kilo!193raymond-ndibetest_1.28main
Show related patches Customize query in GitLab

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Nov 4 2024, 2:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:37:47Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:37:52Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:38 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:38:58Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:39 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:39:01Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:40 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:40:06Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:40 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:40:09Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:41 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:41:17Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:41 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:41:20Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:42 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:42:31Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:44 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:44:28Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:45 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:45:28Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:45 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:45:31Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:46 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:46:27Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:46 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:46:30Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 (T362867)

Stashbot added a comment.Nov 4 2024, 2:47 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-04T14:47:30Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 (T362867)

fnegri removed a parent task: T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29.Nov 4 2024, 4:03 PM
fnegri mentioned this in T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29.
LucasWerkmeister subscribed.Nov 4 2024, 5:33 PM

In T378976: Lexeme-forms on Toolforge returns error, a webservice was dead for a few hours until I manually stopped and started it; might be related to this upgrade? (The startup probe continuously got connect: connection refused, apparently.)

fnegri added a comment.Nov 5 2024, 11:30 AM

might be related to this upgrade?

Potentially yes. Looking at the logs you pasted in T378976, my very vague and unhelpful theory is that it was failing to connect to NFS and access the /data/project/ folder. Manually stopping and starting it probably moved the pod to a different worker node.

Unfortunately I don't think we retain the info on which nodes a pod was scheduled on in the past, if it happens again you can try kubectl get pods -o wide to get the name of the worker node and I can check if there's any NFS issue on that node.

dcaro added a comment.Nov 6 2024, 1:59 PM
In T362867#10292196, @fnegri wrote:

might be related to this upgrade?

Potentially yes. Looking at the logs you pasted in T378976, my very vague and unhelpful theory is that it was failing to connect to NFS and access the /data/project/ folder. Manually stopping and starting it probably moved the pod to a different worker node.

Unfortunately I don't think we retain the info on which nodes a pod was scheduled on in the past, if it happens again you can try kubectl get pods -o wide to get the name of the worker node and I can check if there's any NFS issue on that node.

Found a stray k8s worker that was having trouble with NFS, but was not showing up on the prometheus stats (and the alerts) due to it being in 'confirm migration' status, see T379139: [infra,k8s] node tools-k8s-worker-nfs-24 stopped reporting processes in D state, that might have been one of the issues you were seeing jobs from getting stuck, let's keep an eye in case there's more, but might be sorted now

JeanFred mentioned this in T379452: wikiloves inaccessible since November 4th.Nov 9 2024, 8:13 AM
CodeReviewBot added a project: Patch-For-Review.Nov 10 2024, 2:11 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585

[calico] update kubeVersion

Stashbot added a comment.Nov 10 2024, 2:47 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-10T02:47:11Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T362867)

Stashbot added a comment.Nov 10 2024, 2:47 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-10T02:47:16Z] <raymond-ndibe@cloudcumin1001> Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 (T362867)

Stashbot added a comment.Nov 10 2024, 2:47 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-10T02:47:22Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T362867)

CodeReviewBot added a comment.Nov 10 2024, 3:09 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586

[toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0

CodeReviewBot added a comment.Nov 10 2024, 3:49 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/48

[envvars-api] k8s.io deps to 0.28.14

CodeReviewBot added a comment.Nov 10 2024, 4:09 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/17

[registry-admission] k8s.io deps to 0.28.14

CodeReviewBot added a comment.Nov 10 2024, 4:14 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/14

[ingress-admission] k8s.io deps to 0.28.14

Raymond_Ndibe updated the task description. (Show Details)Nov 10 2024, 4:21 AM
Raymond_Ndibe updated the task description. (Show Details)
CodeReviewBot added a comment.Nov 18 2024, 8:21 AM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193

[lima-kilo] test k8s 1.28 upgrade

CodeReviewBot added a comment.Nov 18 2024, 8:47 AM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/48

[envvars-api] k8s.io deps to 0.28.14

CodeReviewBot added a comment.Nov 18 2024, 8:47 AM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/17

[registry-admission] k8s.io deps to 0.28.14

CodeReviewBot added a comment.Nov 19 2024, 9:07 AM

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/602

ingress-admission: bump to 0.0.54-20241119090358-300b9ae5

CodeReviewBot added a comment.Nov 19 2024, 7:09 PM

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/608

registry-admission: bump to 0.0.54-20241119190632-ec34b7a8

CodeReviewBot added a comment.Nov 19 2024, 7:10 PM

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/609

envvars-api: bump to 0.0.62-20241119190711-285eda5c

Stashbot added a comment.Nov 19 2024, 7:16 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:16:23Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:17 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:17:32Z] <raymond-ndibe@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:17 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:17:45Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:23 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:23:19Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:23 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:23:58Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:24 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:24:13Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:30:12Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:30:46Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:32 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:32:16Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:32 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:32:52Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:37:45Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:38 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:38:12Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:41 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:41:41Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362867)

Stashbot added a comment.Nov 19 2024, 7:47 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:47:22Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362867)

CodeReviewBot added a comment.Nov 19 2024, 7:53 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/608

registry-admission: bump to 0.0.54-20241119190632-ec34b7a8

Stashbot added a comment.Nov 19 2024, 7:54 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T19:54:31Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T362867)

Stashbot added a comment.Nov 19 2024, 8:00 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:00:17Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T362867)

Stashbot added a comment.Nov 19 2024, 8:01 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:01:40Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T362867)

CodeReviewBot added a comment.Nov 19 2024, 8:02 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/602

ingress-admission: bump to 0.0.54-20241119090358-300b9ae5

Stashbot added a comment.Nov 19 2024, 8:04 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:04:05Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component calico (T362867)

Stashbot added a comment.Nov 19 2024, 8:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:07:52Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T362867)

Stashbot added a comment.Nov 19 2024, 8:09 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:09:45Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T362867)

LucasWerkmeister unsubscribed.Nov 19 2024, 8:11 PM
Stashbot added a comment.Nov 19 2024, 8:12 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:12:15Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component calico (T362867)

CodeReviewBot added a comment.Nov 19 2024, 8:12 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586

[toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0

Stashbot added a comment.Nov 19 2024, 8:14 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:14:23Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component envvars-api (T362867)

Stashbot added a comment.Nov 19 2024, 8:17 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:17:07Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T362867)

Stashbot added a comment.Nov 19 2024, 8:20 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:20:24Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api (T362867)

Stashbot added a comment.Nov 19 2024, 8:28 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:28:02Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component envvars-api (T362867)

CodeReviewBot added a comment.Nov 19 2024, 8:28 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585

[toolforge-deploy] update calico kubeVersion

Stashbot added a comment.Nov 19 2024, 8:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:30:03Z] <raymond-ndibe@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api (T362867)

Stashbot added a comment.Nov 19 2024, 8:31 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:31:14Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component envvars-api (T362867)

Stashbot added a comment.Nov 19 2024, 8:31 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-19T20:31:18Z] <raymond-ndibe@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api (T362867)

CodeReviewBot added a comment.Nov 19 2024, 9:19 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/609

envvars-api: bump to 0.0.62-20241119190711-285eda5c

CodeReviewBot added a comment.Nov 19 2024, 11:16 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/21

[volume-admission] k8s.io deps to 0.28.14

CodeReviewBot added a comment.Nov 20 2024, 12:04 AM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/21

[volume-admission] k8s.io deps to 0.28.14

CodeReviewBot added a comment.Nov 20 2024, 12:06 AM

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/610

volume-admission: bump to 0.0.59-20241120000415-71c39564

Stashbot added a comment.Nov 20 2024, 12:09 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-20T00:09:39Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T362867)

Stashbot added a comment.Nov 20 2024, 12:15 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-20T00:15:31Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T362867)

Stashbot added a comment.Nov 20 2024, 12:16 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-20T00:16:18Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T362867)

Stashbot added a comment.Nov 20 2024, 12:22 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-11-20T00:22:49Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T362867)

CodeReviewBot added a comment.Nov 20 2024, 12:23 AM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/610

volume-admission: bump to 0.0.59-20241120000415-71c39564

Raymond_Ndibe updated the task description. (Show Details)Nov 20 2024, 12:24 AM
dcaro closed subtask T378500: [components-api] Add feature flag to disable user endpoints for deployment in tools as Resolved.Nov 25 2024, 5:10 PM
dcaro moved this task from In Review to In Progress on the Toolforge (Toolforge iteration 16) board.Nov 26 2024, 3:08 PM
dcaro edited projects, added Toolforge (Toolforge iteration 17); removed Toolforge (Toolforge iteration 16).Jan 8 2025, 12:49 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 17) board.
fnegri edited projects, added cloud-services-team (FY2024/2025-Q3-Q4); removed cloud-services-team.Jan 21 2025, 11:00 AM

What's left before this task can be resolved? Only T370245: [infra,k8s] remove deprecated kubelet flags before 1.28 upgrade (we might be able to remove all custom ones) or also something else?

Raymond_Ndibe moved this task from In Progress to In Review on the Toolforge (Toolforge iteration 17) board.Jan 24 2025, 1:35 PM
Raymond_Ndibe changed the status of subtask T370245: [infra,k8s] remove deprecated kubelet flags before 1.28 upgrade (we might be able to remove all custom ones) from Open to In Progress.
Raymond_Ndibe moved this task from In Review to In Progress on the Toolforge (Toolforge iteration 17) board.
fnegri closed subtask T370245: [infra,k8s] remove deprecated kubelet flags before 1.28 upgrade (we might be able to remove all custom ones) as Resolved.Feb 20 2025, 1:01 PM
fnegri closed this task as Resolved.Feb 20 2025, 1:07 PM
fnegri removed a subtask: T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27.
fnegri moved this task from In Progress to Done on the Toolforge (Toolforge iteration 17) board.

We've been running 1.28 for a while, and all subtasks are now resolved, I'm resolving this one as well.

fnegri moved this task from Backlog to Done on the cloud-services-team (FY2024/2025-Q3-Q4) board.Feb 20 2025, 1:08 PM
Raymond_Ndibe updated the task description. (Show Details)Feb 20 2025, 6:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits