Upgrade all third-party Toolforge Kubernetes components to versions supporting Kubernetes 1.22
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	Nov 5 2021, 7:25 PM

Description

Kubernetes 1.22 removes a bunch of stuff. We also have a significant amount of third-party admin components installed in the cluster. Those all likely need updates.

https://grafana.wmcloud.org/d/dVVFcEAVz/deprecated-kubernetes-api-calls?orgId=1&var-cluster=prometheus-tools&var-versions=1.22

calico
- should be good here per latest upgrade T292698
ingress-nginx
- current version we're using (1.0.x) claims to support up to 1.22

~~jupyterhub?~~
- not relevant anymore as paws is managed separately

metrics components:

kube-state-metrics
- https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics/-/merge_requests/1 should be good at least to 1.23
metrics-server
cadvisor
prometheus

Details

	Subject	Repo	Branch	Lines +/-
	kubeadm: Update kube-state-metrics to 2.2.4	operations/puppet	production	+23 -17

Customize query in gerrit

	Title	Reference	Author	Source Branch	Dest Branch
	Update metrics-server and overhaul its deployment	repos/cloud/toolforge/wmcs-k8s-metrics!2	taavi	update-metrics-server	main
	Upgrade kube-state-metrics to 2.2.4	repos/cloud/toolforge/wmcs-k8s-metrics!1	taavi	kube-state-metrics	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Open	None	T362869 Upgrade Toolforge to Uwubernetes
Open	None	T362868 Upgrade Toolforge Kubernetes to version 1.29
Open	None	T362867 Upgrade Toolforge Kubernetes to version 1.28
Open	None	T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
Open	None	T327025 Upgrade Toolforge Kubernetes to version 1.26
Open	None	T316107 Upgrade Toolforge Kubernetes to version 1.25
Resolved	aborrero	T307651 Upgrade Toolforge Kubernetes to version 1.24
Resolved	taavi	T298005 Upgrade Toolforge Kubernetes to version 1.23
Resolved	taavi	T286856 Upgrade Toolforge Kubernetes to latest 1.22
Resolved	taavi	T295190 Upgrade all third-party Toolforge Kubernetes components to versions supporting Kubernetes 1.22

Event Timeline

taavi created this task.Nov 5 2021, 7:25 PM

Change 740323 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] kubeadm: Update kube-state-metrics to 2.2.4

https://gerrit.wikimedia.org/r/740323

gerritbot added a project: Patch-For-Review.Nov 21 2021, 9:50 AM

taavi updated the task description. (Show Details)Apr 16 2022, 5:31 PM

Change 740323 abandoned by Majavah:

[operations/puppet@production] kubeadm: Update kube-state-metrics to 2.2.4

Reason:

https://gerrit.wikimedia.org/r/740323

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2022, 6:30 PM

taavi updated the task description. (Show Details)Apr 16 2022, 6:57 PM

taavi updated the task description. (Show Details)

taavi updated the task description. (Show Details)Apr 16 2022, 7:00 PM

MR for kube-state-metrics: https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics/-/merge_requests/1

taavi updated the task description. (Show Details)Apr 28 2022, 6:48 AM

taavi updated the task description. (Show Details)May 11 2022, 7:03 PM

fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:45 PM

fnegri moved this task from Kanban to Inbox on the cloud-services-team board.

taavi updated the task description. (Show Details)Feb 9 2023, 11:14 AM

taavi updated the task description. (Show Details)Feb 14 2023, 7:52 PM

Had to do a bit of trickery to figure out the source of the remaining ingress API calls:

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"887d9bba-8c48-48ee-a80c-4757ba645a8a","stage":"ResponseComplete","requestURI":"/apis/extensions/v1beta1/ingresses?allowWatchBookmarks=true\u0026resourceVersion=444308692\u0026timeout=5m40s\u0026timeoutSeconds=340\u0026watch=true","verb":"watch","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["172.16.6.136"],"userAgent":"kube-controller-manager/v1.21.8 (linux/amd64) kubernetes/4a3b558/shared-informers","objectRef":{"resource":"ingresses","apiGroup":"extensions","apiVersion":"v1beta1"},"responseStatus":{"metadata":{},"status":"Success","message":"Connection closed early","code":200},"requestReceivedTimestamp":"2023-02-14T20:11:59.557788Z","stageTimestamp":"2023-02-14T20:17:39.558597Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\"","k8s.io/deprecated":"true","k8s.io/removed-release":"1.22"}}