During the incident T367348: Incident: 2024-06-12 toolforge k8s control plane we had the theory that our kyverno policies (which are namespaced) could somehow be evaluated for resources outside their namespace, creating excessive load.
Given the amount of policies we have (3.5k policies with 2 rules each, total of 7k rules), this load could be exponential and fatal to the cluster.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T362869 [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) | |||
| Open | None | T362868 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 | |||
| Open | None | T362867 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 | |||
| Open | None | T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 | |||
| Open | None | T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26 | |||
| Open | None | T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25 | |||
| In Progress | aborrero | T279110 [infra] Replace PodSecurityPolicy in Toolforge Kubernetes | |||
| In Progress | aborrero | T364297 [k8s,infra] track PSP migration plan | |||
| Resolved | Andrew | T367348 Incident: 2024-06-12 toolforge k8s control plane | |||
| Resolved | aborrero | T367350 [k8s,infra] Verify that kyverno policies are evaluated only for namespaced resources |
In the upstream docs:
Policies can be defined as cluster-wide resources (using the kind ClusterPolicy) or namespaced resources (using the kind Policy). As expected, namespaced policies will only apply to resources within the namespace in which they are defined while cluster-wide policies are applied to matching resources across all namespaces. Otherwise, there is no difference between the two types.
source: https://kyverno.io/docs/kyverno-policies/
With this information, the theory in this ticket loses some weight. But I'll keep the ticket open while I do additional research. I'm also considering to contact upstream about this theory.