Upgrade Toolforge Kubernetes to version 1.23
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• taavi
	Dec 19 2021, 8:57 AM

Description

Kubernetes 1.23 is out: https://kubernetes.io/blog/2021/12/07/kubernetes-1-23-release-announcement/

Last upgrade notes: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes/1.21_to_1.22_notes

Details

	Subject	Repo	Branch	Lines +/-
	aptrepo: drop k8s 1.22 components	operations/puppet	production	+3 -14
	aptrepo: Add kubeadm 1.23 component	operations/puppet	production	+14 -3

Customize query in gerrit

	Title	Reference	Author	Source Branch	Dest Branch
	Update to k8s 1.23	repos/cloud/toolforge/lima-kilo!54	taavi	taavi/k8s-1-23	main

Customize query in GitLab

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T364293 [infra,k8s] Move to kubernetes PAVs and drop kyverno
Open		aborrero	T362869 [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30)
Open		fnegri	T362868 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29
Open		Raymond_Ndibe	T362867 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28
Open		dcaro	T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
Open		Slst2020	T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26
Resolved		aborrero	T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25
Resolved		aborrero	T307651 Upgrade Toolforge Kubernetes to version 1.24
Resolved		• taavi	T298005 Upgrade Toolforge Kubernetes to version 1.23
Resolved		• taavi	T286856 Upgrade Toolforge Kubernetes to latest 1.22
Resolved		• taavi	T280436 Migrate Toolforge admission webhooks to use the admissionregistration.k8s.io/v1 API
Resolved		• taavi	T286857 Update maintain-kubeusers to certificates/v1 api
Resolved		• Bstorm	T289390 Certificate generation is broken in toolsbeta
Resolved		• taavi	T292238 Figure out certificate generation for admission webhooks before we lose the certificates/v1beta1
Resolved		• taavi	T329453 Deploy cert-manager to Toolforge
Resolved		• taavi	T292706 tools-webservice should use ingress v1
Resolved		• taavi	T292771 upgrade to ingress-nginx 1.0
Resolved		rook	T291589 Upgrade paws jupyterhub
Resolved		• taavi	T295190 Upgrade all third-party Toolforge Kubernetes components to versions supporting Kubernetes 1.22
Resolved		• taavi	T306296 Ensure k8s-status tool will work on Kubernetes 1.22
Resolved		rook	T308172 Upgrade PAWS to Kubernetes 1.21
Resolved		• taavi	T282942 Upgrade Toolforge Kubernetes to latest 1.21
Resolved		rook	T280402 Upgrade Toolforge Kubernetes to latest 1.20
Resolved		• taavi	T280340 Upgrade Toolforge Kubernetes to latest 1.19
Resolved		• taavi	T280299 Upgrade Toolforge Kubernetes to latest 1.18
Resolved		• taavi	T280300 Validate that maintain-kubeusers works with k8s 1.18
Resolved		• taavi	T280301 Refresh certs that are not controlled by kubeadm
Resolved		• taavi	T280302 Upgrade PAWS Kubernetes to the latest 1.18 release
Resolved		• taavi	T264221 Upgrade the nginx ingress controller in Toolforge (and likely PAWS)
Resolved		• Bstorm	T282087 Support cinder or expanded ephemeral disk worker nodes on Toolforge Kubernetes
Resolved		• taavi	T284353 Fix prometheus monitoring for Toolforge Ingress
Resolved		• taavi	T280342 Upgrade Calico to 3.18
Resolved		• taavi	T280360 Update ingress-admission-controller to support Ingress from networking.k8s.io/v1
Resolved		• taavi	T279106 Establish replacement for PodPresets in Toolforge Kubernetes
Resolved		• Bstorm	T292672 Issue creating pods after migration away from PodPresets
Declined		None	T290531 Weird issue with the wmcs-k8s-node-upgrade.py script
Resolved		rook	T291976 Upgrade toolsbeta to k8s 1.20
Resolved		• taavi	T292698 Upgrade calico to 3.21
Resolved	BUG REPORT	None	T308189 Toolforge jobs stopped getting scheduled around the same time as the Toolforge k8s cluster upgrade
Open		None	T308203 Create an alert for K8s cronjobs
Resolved		• taavi	T308204 toolforge-jobs should set startingDeadlineSeconds by default
Resolved		• taavi	T308205 Re-enable CronJobControllerV2
Open		None	T308381 toolforge: Scrape Kubernetes controller-manager and apiserver metrics into Prometheus
Resolved		rook	T308274 test PAWS in k8s 1.21.8
Resolved		• taavi	T329869 Remove tool RBAC rules for APIs removed in Kubernetes 1.22
Resolved	BUG REPORT	• taavi	T331572 maintain-kubeusers container in CrashLoopBackoff preventing new tool creation after 'user-maintainer' ClusterRole changes
Resolved		• taavi	T331619 toolforge: rbac: change existing roles to reference PSP in the policy group
Resolved		• taavi	T334380 Toolforge k8s: kube-scheduler permissions errors
Resolved		• taavi	T334382 Toolforge k8s: kube-controller-manager permission errors
Resolved		• taavi	T329677 Upgrade Calico to 3.24

Event Timeline

• taavi created this task.Dec 19 2021, 8:57 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 19 2021, 8:57 AM

• taavi added a subtask: T286856: Upgrade Toolforge Kubernetes to latest 1.22.Dec 19 2021, 8:57 AM

• taavi added a project: cloud-services-team (Kanban).

• BilalShirwani closed this task as a duplicate of T298058: [Impact Analysis] Evaluate Impact of Mobile Reply and New Discussion Tools.Dec 21 2021, 12:02 AM

JJMC89 reopened this task as Open.Dec 21 2021, 12:06 AM

• taavi added a parent task: T307651: Upgrade Toolforge Kubernetes to version 1.24.May 5 2022, 6:58 AM

fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:39 PM

fnegri moved this task from Kanban to Inbox on the cloud-services-team board.

• taavi closed subtask T286856: Upgrade Toolforge Kubernetes to latest 1.22 as Resolved.Apr 10 2023, 12:06 PM

• taavi closed subtask T329677: Upgrade Calico to 3.24 as Resolved.Apr 11 2023, 9:52 AM

Change 908276 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] aptrepo: Add kubeadm 1.23 component

https://gerrit.wikimedia.org/r/908276

gerritbot added a project: Patch-For-Review.Apr 12 2023, 3:45 PM

Change 908276 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] aptrepo: Add kubeadm 1.23 component

https://gerrit.wikimedia.org/r/908276

Maintenance_bot removed a project: Patch-For-Review.Apr 13 2023, 3:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-14T08:21:21Z] <arturo> aborrero@apt2001:~ $ sudo -i reprepro --noskipold --component thirdparty/kubeadm-k8s-1-23 update buster-wikimedia (T298005)

I've reviewed the release notes and don't see anything concerning in there.

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/54

Update to k8s 1.23

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/54

Update to k8s 1.23

Maintenance_bot removed a project: Patch-For-Review.Jul 8 2023, 2:10 PM

fnegri edited projects, added cloud-services-team (FY2023/2024-Q1-Q2), Goal; removed cloud-services-team.Jul 26 2023, 3:58 PM

fnegri moved this task from Backlog to Planned Goals on the cloud-services-team (FY2023/2024-Q1-Q2) board.Jul 26 2023, 4:02 PM

aborrero added a project: User-aborrero.Jul 28 2023, 8:50 AM

• taavi claimed this task.Aug 30 2023, 8:14 AM

Mentioned in SAL (#wikimedia-cloud) [2023-08-30T08:15:00Z] <taavi> updating toolsbeta k8s cluster to 1.23 to test new cookbooks, T298005 T343869

Stashbot mentioned this in T343869: Turn wmcs-k8s-node-upgrade.py into a set of cookbooks.Aug 30 2023, 8:15 AM

aborrero moved this task from Backlog to Doing on the User-aborrero board.Aug 30 2023, 10:56 AM

• taavi updated Other Assignee, added: aborrero.Aug 30 2023, 11:01 AM

• taavi moved this task from Backlog to In Progress on the Toolforge board.Aug 30 2023, 11:04 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-08-30T11:15:29Z] <wm-bot2> toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 (T298005) - cookbook ran by arturo@nostromo

Mentioned in SAL (#wikimedia-cloud-feed) [2023-08-30T11:17:17Z] <wm-bot2> toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 (T298005) - cookbook ran by arturo@nostromo

Mentioned in SAL (#wikimedia-cloud-feed) [2023-08-30T11:18:54Z] <wm-bot2> toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 (T298005) - cookbook ran by arturo@nostromo

aborrero moved this task from Doing to Next on the User-aborrero board.Sep 6 2023, 8:48 AM

aborrero moved this task from Next to Radar/observer on the User-aborrero board.Sep 18 2023, 8:50 AM

dcaro updated the task description. (Show Details)Sep 26 2023, 2:03 PM

• taavi updated Other Assignee, removed: aborrero.Oct 17 2023, 10:40 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-18T11:03:50Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.22.17 to 1.23.17 (T298005)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-18T11:04:17Z] <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.22.17 to 1.23.17 (T298005)

• taavi closed this task as Resolved.Oct 18 2023, 12:08 PM

• taavi edited projects, added Toolforge (Toolforge iteration 01); removed User-aborrero, Toolforge.

• taavi moved this task from Next Up to Done on the Toolforge (Toolforge iteration 01) board.

Change 966865 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] aptrepo: drop k8s 1.22 components

https://gerrit.wikimedia.org/r/966865

gerritbot added a project: Patch-For-Review.Oct 18 2023, 1:16 PM

Change 966865 merged by Majavah:

[operations/puppet@production] aptrepo: drop k8s 1.22 components

https://gerrit.wikimedia.org/r/966865

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2023, 10:10 AM

fnegri moved this task from Planned Goals to Done on the cloud-services-team (FY2023/2024-Q1-Q2) board.Nov 16 2023, 6:45 PM