Upgrade Toolforge Kubernetes to version 1.24
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	May 5 2022, 6:57 AM

Description

https://kubernetes.io/blog/2022/05/03/kubernetes-1-24-release-announcement/

Details

Other Assignee: taavi

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	lima-kilo: bump kubernetes version to 1.24	repos/cloud/toolforge/lima-kilo!108	aborrero	arturo-1390-lima-kilo-bump-kub	main

Customize query in GitLab

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Raymond_Ndibe	T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
			Restricted Task
Resolved		Slst2020	T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26
Resolved		• aborrero	T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25
Resolved		• aborrero	T307651 Upgrade Toolforge Kubernetes to version 1.24
Resolved		taavi	T298005 Upgrade Toolforge Kubernetes to version 1.23
Resolved		taavi	T286856 Upgrade Toolforge Kubernetes to latest 1.22
Resolved		taavi	T280436 Migrate Toolforge admission webhooks to use the admissionregistration.k8s.io/v1 API
Resolved		taavi	T286857 Update maintain-kubeusers to certificates/v1 api
Resolved		• Bstorm	T289390 Certificate generation is broken in toolsbeta
Resolved		taavi	T292238 Figure out certificate generation for admission webhooks before we lose the certificates/v1beta1
Resolved		taavi	T329453 Deploy cert-manager to Toolforge
Resolved		taavi	T292706 tools-webservice should use ingress v1
Resolved		taavi	T292771 upgrade to ingress-nginx 1.0
Resolved		• rook	T291589 Upgrade paws jupyterhub
Resolved		taavi	T295190 Upgrade all third-party Toolforge Kubernetes components to versions supporting Kubernetes 1.22
Resolved		taavi	T306296 Ensure k8s-status tool will work on Kubernetes 1.22
Resolved		• rook	T308172 Upgrade PAWS to Kubernetes 1.21
Resolved		taavi	T282942 Upgrade Toolforge Kubernetes to latest 1.21
Resolved		• rook	T280402 Upgrade Toolforge Kubernetes to latest 1.20
Resolved		taavi	T280340 Upgrade Toolforge Kubernetes to latest 1.19
Resolved		taavi	T280299 Upgrade Toolforge Kubernetes to latest 1.18
Resolved		taavi	T280300 Validate that maintain-kubeusers works with k8s 1.18
Resolved		taavi	T280301 Refresh certs that are not controlled by kubeadm
Resolved		taavi	T280302 Upgrade PAWS Kubernetes to the latest 1.18 release
Resolved		taavi	T264221 Upgrade the nginx ingress controller in Toolforge (and likely PAWS)
Resolved		• Bstorm	T282087 Support cinder or expanded ephemeral disk worker nodes on Toolforge Kubernetes
Resolved		taavi	T284353 Fix prometheus monitoring for Toolforge Ingress
Resolved		taavi	T280342 Upgrade Calico to 3.18
Resolved		taavi	T280360 Update ingress-admission-controller to support Ingress from networking.k8s.io/v1
Resolved		taavi	T279106 Establish replacement for PodPresets in Toolforge Kubernetes
Resolved		• Bstorm	T292672 Issue creating pods after migration away from PodPresets
Declined		None	T290531 Weird issue with the wmcs-k8s-node-upgrade.py script
Resolved		• rook	T291976 Upgrade toolsbeta to k8s 1.20
Resolved		taavi	T292698 Upgrade calico to 3.21
Resolved	BUG REPORT	None	T308189 Toolforge jobs stopped getting scheduled around the same time as the Toolforge k8s cluster upgrade
Duplicate		None	T308203 Create an alert for K8s cronjobs
Resolved		taavi	T308204 toolforge-jobs should set startingDeadlineSeconds by default
Resolved		taavi	T308205 Re-enable CronJobControllerV2
Open		None	T308381 [infra,k8s] Scrape Kubernetes controller-manager and apiserver metrics into Prometheus
Resolved		• rook	T308274 test PAWS in k8s 1.21.8
Resolved		taavi	T329869 Remove tool RBAC rules for APIs removed in Kubernetes 1.22
Resolved	BUG REPORT	taavi	T331572 maintain-kubeusers container in CrashLoopBackoff preventing new tool creation after 'user-maintainer' ClusterRole changes
Resolved		taavi	T331619 toolforge: rbac: change existing roles to reference PSP in the policy group
Resolved		taavi	T334380 Toolforge k8s: kube-scheduler permissions errors
Resolved		taavi	T334382 Toolforge k8s: kube-controller-manager permission errors
Resolved		taavi	T329677 Upgrade Calico to 3.24
Resolved		taavi	T284656 Toolforge k8s: Migrate workers to Containerd and Bookworm
Resolved		taavi	T349795 Upgrade cadvisor
Resolved		taavi	T350227 toolforge prometheus servers OOMing
Resolved		taavi	T357901 Request increased server-group-members quota for tools
Resolved		• aborrero	T358476 toolforge k8s: some static pods needs manual restart
Resolved		taavi	T314665 Toolforge: Replace all bastion with grid-less bookworm based bastion hosts
Resolved		taavi	T314664 [infra] Decommission the Grid Engine infrastructure
Resolved		komla	T313405 Migrate remaining tools off Gridengine
Declined		None	T316124 Make Grid Engine tooling emit deprecation warnings
Resolved		taavi	T329619 Toolforge: decide what to do with tools that have 'stretch' in their service manifests
Resolved		Andrew	T353351 Provide tools for disabling the grid for specific tools
Resolved		Andrew	T353642 Relocate disable-tool-archive-dbs.service
Resolved		taavi	T358333 Remove toolschecker grid engine checks
Resolved		taavi	T359934 [infra] Archive grid engine related infrastructure tools
Resolved		taavi	T359935 [infra] Archive grid engine related Gerrit repositories
Stalled		dcaro	T356377 [toolforge] simplify calling the different toolforge apis from within the containers
Stalled		None	T356262 [jobs-cli,builds-cli,toolforge-cli,webservice] Consolidate the Toolforge CLIs
Open		None	T356261 [toolforge-cli,jobs-cli,builds-cli,envvars-cli] Explore OpenAPI SDK tooling for client consolidation
Open		None	T398651 [toolforge-cli-gen] review the https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-gen-cli client as potential consolidation
Resolved		Slst2020	T354745 [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client
Resolved		Slst2020	T358100 [toolforge API] expose all backend APIs OpenAPI specs
Resolved		• aborrero	T356523 [jobs-api] introduce OpenAPI to jobs framework
Resolved		Slst2020	T362299 [api-gateway] Add a python server to serve consolidated openapi docs
Declined		None	T369693 [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images
Resolved	BUG REPORT	dcaro	T369569 `webservice` requires effective user to be the tool user and listed in NSS passwd data
Resolved	BUG REPORT	dcaro	T369573 `toolforge jobs` requires current user to be the tool user and listed in NSS passwd data
Resolved	BUG REPORT	None	T360488 Missing Perl packages on dev.toolforge.org for anomiebot workflows
Resolved		dcaro	T371556 Please re-install "joe"
Resolved		taavi	T397459 Lock down tools-sgebastion-10 (login-buster.toolforge.org) to only members of tools with known dependencies on it
Resolved		• aborrero	T359619 toolforge: prepare deb packages for k8s 1.24
Resolved		• aborrero	T359620 toolforge: upgrade k8s etcd nodes to debian bookworm
Resolved		• aborrero	T359638 [toolsbeta,infra] upgrade kubernetes to 1.24
Resolved		• aborrero	T359639 toolforge: verify etcd version is compatible with k8s 1.24
Resolved		• aborrero	T359798 refresh kube-state-metrics version for k8s 1.24
Resolved		• aborrero	T359927 [cookbook] wmcs.toolforge.k8s.prepare_upgrade: be more flexible checking for deb package components

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:48:14Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:48:17Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:49:15Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:49:19Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:50:21Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:50:26Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:51:24Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:51:27Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:52:29Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:52:32Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:53:35Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:53:38Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:54:38Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:54:41Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T11:55:43Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:05:17Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:11:18Z] <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:11:22Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:15:39Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:15:42Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:18:13Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:18:17Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:19:10Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:19:13Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:20:17Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:20:20Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:21:19Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:21:22Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:22:26Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:22:29Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:23:31Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:23:36Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:24:40Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:24:43Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:25:45Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:25:48Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:26:47Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:26:50Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:27:49Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:27:53Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:28:53Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:28:58Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:29:56Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:29:59Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:31:03Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:31:06Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:32:07Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:32:10Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:33:08Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:33:12Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:34:12Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:34:17Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:35:17Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:35:20Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:36:24Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:46:21Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:47:23Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:47:27Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:48:28Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:48:31Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:49:36Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:49:40Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:50:41Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:50:45Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:51:42Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:51:46Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:52:47Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:52:53Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:53:51Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:53:55Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:54:57Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:55:01Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:56:09Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:56:12Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:57:14Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:57:18Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:58:16Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:58:21Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:59:23Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T12:59:26Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:00:30Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:00:34Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:01:35Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:01:38Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:02:42Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:02:45Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:03:54Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:11:02Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:12:06Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:12:09Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:13:14Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:13:17Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 (T307651)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-18T13:14:34Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 (T307651)