During docker stress testing we had multiple issues with kubernetes nodes (the ganeti ones) failing completely because calico-node was evicted from them. T289111 describes the aftermath of that exact situation (which stayed undetected mainly because eqiad is depooled).
Enabling the admission controller is the easy task but we will also want to limit the Kubernetes default priority classes `system-cluster-critical` and `system-node-critical` to only be used for Pods in namespaces we "trust" (like `kube-system` for services clusters, ml may need additional for istio, kf*).
This can be done by providing AdmissionConfiguration via kube-apiserver flag `--admission-control-config-file` like:
- name: "ResourceQuota"
- resource: pods
- scopeName: PriorityClass
And explicitly granting namespaces the permission to use those classes by adding a ResourceQuota object:
- operator : In