the plan could be this:

• finish T362872: Decision Request - Toolforge policy agent enforcement model
• finish T362050: toolforge: review pod templates for PSP replacement
• finish T364113: toolforge: identify and cache in our container registry all kyverno images
• deploy kyverno (with policies in audit mode) -- https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/238
• deploy updated maintain-kubeusers -- https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18
• mutate existing workloads to add missing pod template elements (!)
• re-deploy kyverno (with policies in validation mode)
• drop PSP

if we don't want to mutate existing workloads (!) by hand (script or something) we will need to carefully craft the kyverno policies so the templates produced in the PSP days are valid on kyverno's eyes.

Updated T362050: toolforge: review pod templates for PSP replacement to make sure our pod templates are updated accordingly.

Another point to consider: how to back-fill per-tool kyverno policies for existing tools.

updated plan:

• finish T362872: Decision Request - Toolforge policy agent enforcement model
• finish T362050: toolforge: review pod templates for PSP replacement
• finish T364113: toolforge: identify and cache in our container registry all kyverno images
• deploy refactored maintain-kubeusers -- T364312: [maintain-kubeusers,infra,k8s]: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies)
• deploy kyverno itself -- https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/238
• deploy kyverno policies (with policies in audit mode) -- https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18
• re-deploy kyverno (with policies in validation mode)
• drop PSP

on the latest tests I've conducted, I detected that the kyverno mutation rule somehow clashes with the PSP mutations.

• kyverno mutation rule adding hostIPC: false
• then PSP removing it
• then the kyverno validation rule requiring it being set
• therefore kyverno validation failing

This temporal conflict between the Kyverno pod policy and PSP is something that needs to be carefully tested, and configured, for the cluster to be happy during the -hopefully brief- period in which the 2 mechanisms are alive together.

In T364297#9871364, @aborrero wrote:

on the latest tests I've conducted, I detected that the kyverno mutation rule somehow clashes with the PSP mutations.

• kyverno mutation rule adding hostIPC: false
• then PSP removing it
• then the kyverno validation rule requiring it being set
• therefore kyverno validation failing

This temporal conflict between the Kyverno pod policy and PSP is something that needs to be carefully tested, and configured, for the cluster to be happy during the -hopefully brief- period in which the 2 mechanisms are alive together.

Making optional in the kyverno policy the fields that PSP is deleting works fine!

This should also be a valid solution when we drop PSP.

this has been completed.