Maniphest T367348

Incident: 2024-06-12 toolforge k8s control plane
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Andrew
	Jun 12 2024, 6:51 PM

Tags

Referenced Files

None

Subscribers

Description

System-wide renaming of kyverno policies overloaded tool control nodes rendering API unresponsive

Lack of working api prevented rollback

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Raymond_Ndibe	T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
					Restricted Task
		Resolved		Slst2020	T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26
		Resolved		• aborrero	T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25
		Resolved		• aborrero	T279110 [infra] Replace PodSecurityPolicy in Toolforge Kubernetes
		Resolved		• aborrero	T364297 [k8s,infra] track PSP migration plan
		Resolved		Andrew	T367348 Incident: 2024-06-12 toolforge k8s control plane
		Duplicate		dcaro	T367349 Fix HA proxy load-balancer health check monitor to not poll nodes where the API is not responding
		Resolved		• aborrero	T367350 [k8s,infra] Verify that kyverno policies are evaluated only for namespaced resources
		Resolved		• aborrero	T367386 [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways
		Resolved		• aborrero	T367388 [k8s,infra] consider scaling the k8s control plane
		Resolved		• aborrero	T367389 [k8s,infra,alerting] improve HAproxy and k8s apiserver interaction

Event Timeline

Andrew created this task.Jun 12 2024, 6:51 PM

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

Andrew added a subtask: T333934: [k8s,infra] scale up coredns replicas.Jun 12 2024, 6:52 PM

Andrew edited projects, added Toolforge; removed Cloud-Services.

• aborrero added a project: User-aborrero.Jun 13 2024, 8:56 AM

• aborrero mentioned this in T364297: [k8s,infra] track PSP migration plan.

• aborrero added a parent task: T364297: [k8s,infra] track PSP migration plan.

• aborrero subscribed.

• aborrero mentioned this in T367350: [k8s,infra] Verify that kyverno policies are evaluated only for namespaced resources.Jun 13 2024, 9:03 AM

• aborrero triaged this task as High priority.Jun 13 2024, 9:05 AM

• aborrero changed the status of subtask T367350: [k8s,infra] Verify that kyverno policies are evaluated only for namespaced resources from Open to In Progress.

• aborrero changed the status of subtask T367386: [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways from Open to In Progress.Jun 13 2024, 9:13 AM

• aborrero changed the status of subtask T367388: [k8s,infra] consider scaling the k8s control plane from Open to In Progress.Jun 13 2024, 9:19 AM

• aborrero mentioned this in T367389: [k8s,infra,alerting] improve HAproxy and k8s apiserver interaction.Jun 13 2024, 9:34 AM

• aborrero closed subtask T367350: [k8s,infra] Verify that kyverno policies are evaluated only for namespaced resources as Resolved.Jun 17 2024, 11:56 AM

• aborrero mentioned this in T367950: Decision Request - Toolforge pod security via custom admission webhook.Jun 19 2024, 8:47 AM

• aborrero closed subtask T367386: [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways as Resolved.Jun 19 2024, 8:55 AM

• aborrero reopened subtask T367386: [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways as In Progress.Jun 20 2024, 8:48 AM

• aborrero closed subtask T367389: [k8s,infra,alerting] improve HAproxy and k8s apiserver interaction as Resolved.Jun 20 2024, 12:30 PM

• aborrero closed subtask T367388: [k8s,infra] consider scaling the k8s control plane as Resolved.Jun 20 2024, 12:39 PM

• aborrero closed subtask T367386: [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways as Resolved.Jun 20 2024, 1:13 PM

everything done here.

see https://wikitech.wikimedia.org/wiki/Incidents/2024-06-12_WMCS_toolforge_k8s_control_plane