toolforge: kyverno: change policies to Enforce
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aborrero
	Jun 21 2024, 11:39 AM

Description

We initially deployed Toolforge Kyverno pod security policies in Audit mode to evaluate how they operate in our cluster.

We need to change them to Enforce if we want to move forward with T279110: [infra] Replace PodSecurityPolicy in Toolforge Kubernetes.

Related Objects
Search...

Status	Assigned	Task
In Progress	Raymond_Ndibe	T362867 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28
Resolved	Raymond_Ndibe	T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
		Restricted Task
Resolved	Slst2020	T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26
Resolved	aborrero	T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25
Resolved	aborrero	T279110 [infra] Replace PodSecurityPolicy in Toolforge Kubernetes
Resolved	aborrero	T364297 [k8s,infra] track PSP migration plan
Resolved	aborrero	T368141 toolforge: kyverno: change policies to Enforce

Event Timeline

aborrero created this task.Jun 21 2024, 11:39 AM

aborrero mentioned this in T364297: [k8s,infra] track PSP migration plan.

aborrero changed the task status from Open to In Progress.Jun 21 2024, 11:41 AM

aborrero triaged this task as High priority.

aborrero moved this task from Backlog to Doing on the User-aborrero board.

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/46

kyverno_pod_policy: set validation to Enforce

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/47

kyverno_pod_policy: use patch operation in do_update()

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/47

kyverno_pod_policy: use patch operation in do_update()

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/342

maintain-kubeusers: bump to 0.0.153-20240624092755-fd0244da

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/342

maintain-kubeusers: bump to 0.0.153-20240624092755-fd0244da

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/343

kyverno: re-enable the reports controller

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/343

kyverno: re-enable the reports controller

Question: how setting the policy to enforce will affect existing deployments that were created prior to the changes in T362050: toolforge: review pod templates for PSP replacement?

Well, per upstream docs:

Background scanning, enabled by default in a Policy or ClusterPolicy object with the spec.background field, allows Kyverno to periodically scan existing resources and find if they match any validate or verifyImages rules. If existing resources are found which would violate an existing policy, the background scan notes them in a ClusterPolicyReport or a PolicyReport object, depending on if the resource is namespaced or not. It does not block any existing resources that match a rule, even in Enforce mode. It has no effect on either generate or mutate rules for the purposes of reporting.

source: https://kyverno.io/docs/policy-reports/background/

Also:

background scanning and is enabled by default unless spec.background is set to false in a policy

Things I've checked:

re-read the upstream docs about what happens if you set policies to enforce while there are offending resources defined in the cluster. This is the case of webservices defined by older versions of the CLI (same for jobs). See previous comment https://phabricator.wikimedia.org/T368141#9921404
performance impact of changing thousand of policies from Audit to Enforce. I tested this in lima-kilo in my laptop. I did not see any relevant impact.
templates generated by newer jobs/webservices are valid for the new policies. Tested as part of T362050: toolforge: review pod templates for PSP replacement
functional tests passes pre/post changes, in lima-kilo

I plan to perform this change tomorrow wednesday 2024-06-26 at 08:30Z

Before setting policies to Enforce, I've checked again the policy reports.

There are a bunch of policy violations:

aborrero@tools-k8s-control-7:~$ sudo -i kubectl get event -A | grep PolicyViolation  | wc -l
11760

All of them contain the same content, an autogen rule complaining about the runAsGroup parameter:

tool-zumraband                              33m         Warning   PolicyViolation           deployment/zumraband                                             policy toolforge-kyverno-pod-policy/autogen-toolforge-validate-pod-policy fail: validation error: pod security configuration must be correct. rule autogen-toolforge-validate-pod-policy failed at path /spec/template/spec/securityContext/runAsGroup/

I think this happens because Deployment and other pod-generators that were created by old version of jobs-cli and webservice don't inject the securityContext. They were relying on PSP mutating the Pod resources.

All these PolicyViolations are from autogen rules. They are not for Pod resources, but for Pod-generators, like Deployments. Pod resources have a good securityContext, either by the PSP mutation, or by the new Kyverno mutation.

This means that if we set policies to Enforce, Pods being created will all pass the validations.

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/46

kyverno_pod_policy: set validation to Enforce

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/351

maintain-kubeusers: bump to 0.0.155-20240626090852-f6b198f9

Mentioned in SAL (#wikimedia-cloud) [2024-06-26T09:15:41Z] <arturo> setting kyverno policies to Enforce (T368141)