Incident: 2024-06-12 toolforge k8s control plane
Open, HighPublic
Actions

Assigned To

Authored By

	Andrew
	Wed, Jun 12, 6:51 PM

Description

System-wide renaming of kyverno policies overloaded tool control nodes rendering API unresponsive

Lack of working api prevented rollback

Status	Assigned	Task
Open	None	T362869 [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30)
Open	None	T362868 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29
Open	None	T362867 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28
Open	None	T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
Open	None	T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26
Open	None	T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25
In Progress	aborrero	T279110 [infra] Replace PodSecurityPolicy in Toolforge Kubernetes
In Progress	aborrero	T364297 [k8s,infra] track PSP migration plan
Open	Andrew	T367348 Incident: 2024-06-12 toolforge k8s control plane
Open	None	T333934 [k8s,infra] scale up coredns replicas
Open	None	T239404 [k8s,infra] evaluate DNS (coredns) autoscale options
Duplicate	dcaro	T367349 Fix HA proxy load-balancer health check monitor to not poll nodes where the API is not responding
Resolved	aborrero	T367350 [k8s,infra] Verify that kyverno policies are evaluated only for namespaced resources
In Progress	None	T367386 [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways
In Progress	aborrero	T367388 [k8s,infra] consider scaling the k8s control plane
Open	None	T367389 [k8s,infra,alerting] improve HAproxy and k8s apiserver interaction

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

Andrew edited projects, added Toolforge; removed Cloud-Services.

aborrero subscribed.

aborrero triaged this task as High priority.Thu, Jun 13, 9:05 AM