Pod Security Policies (PSP), starting with the Kubernetes 1.21, will begin the process of deprecation with the intention to fully remove it in a future release. ...
Full blog post draft here)
Github issue at: https://github.com/kubernetes/kubernetes/pull/97171
While we started implementing PSPs in T228967, they never fully made it to our clusters (as of k8s <1.16).
With Kubernetes 1.16 upgrades we want to implement the recommended restrictions as far as possible without to much effort (that we might have to re-spend with the deprecation). Although there are alternative options around currently, we still have some time and it can be assumed that those options evolve in the near future and we can migrate off of PSPs at a later point.
Things we currently enforce via PSPs (which a replacement needs to provide as well):
- Prohibit running privileged containers, hostIPC, hostPID, hostNetwork
- Ensure containers run as non-root
- Restrict the use of volumes (only specific volume plugins, only specific host paths)
- Prohibit containers with fsGroup/supplementalGroup of 0
- Ensure capabilities are dropped
Apart from the "privileged" policy that effectively allows everything (required for things like calico for example) we only have two PSPs in wikikube. For details about them, see helmfile.d/admin_ng/helmfile_psp.yaml.
WMCS tasks about it:
- T279110: [infra] Replace PodSecurityPolicy in Toolforge Kubernetes
- T335131: [infra,k8s] replace admission controllers with an existing policy admin project
- T362233: Decision Request - Toolforge policy agent
Research about the way forward / alternatives
https://wikitech.wikimedia.org/wiki/User:JMeybohm/PSP_Replacement
Preparation for migrating away from PSPs
Run validation against the restricted PSS profile
- T290020: Enable audit logging for kube-apiserver for wikikube
- Enable restriced PSS profile in audit mode for wikikube https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1018952
Remove seccomp.security.alpha.kubernetes.io/* annotations from PSPs
Current state
Currently (with PSPs) incoming Pods are mutated by the apiserver which adds securityContext.seccompProfile.type: RuntimeDefault to all containers of that pod that have not explicitly set the seccomp profile (controlled by the PSP annotation seccomp.security.alpha.kubernetes.io/defaultProfileName).
This means that effectively all our containers have securityContext.seccompProfile.type: set to either RuntimeDefault or whatever they choose (check with P60658).
The PSP does also validate the securityContext.seccompProfile.type field of all containers against a list of allowed profiles (controlled by the PSP annotation seccomp.security.alpha.kubernetes.io/allowedProfileNames). If the profile selected by a container is not in the list, the Pod is rejected.
Post PSP state
PSA/PSS can't do any mutation. Because of that the need to provide the valid Podspec to the apiserver for validation to pass.
There is a SeccompDefault v1.23 alpha feature | Kubernetes that enforces the runtime/default seccomp profile to all containers on container runtime level, which means the containers do always run with the default seccomp profile but this is not visible in the API - making containers without an explicit seecurityContext.seccompProfile.type fail PSS (restricted) validation. The baseline PSS profile does allow securityContext.seccompProfile.type to be unset.
Path forward
During the migration to PSS we need to allow all seccomp profiles because even if mutation (defaultProfileName) is disabled , the PSP will still validate the seccomp.security.alpha.kubernetes.io/pod annotation (but not securityContext.seccompProfile).
Remove apparmor.security.beta.kubernetes.io/* annotations from PSPs
Current state
This is more or less the same as with seccomp above. The difference is that there is no field in Podspec for the apparmor profile. Instead the apparmor profile is added as a list of annotations (one for each container) to the Pod's annotations (container.apparmor.security.beta.kubernetes.io/<container_name>: runtime/default).
The container runtime does enforce the apparmor profile runtime/default if none is chosen explicitly (like with the SeccompDefault feature, but out-of-the-box), check with P60658.
Post PSP state
Same restrictions apply in term of mutation, but PSS(baseline) and PSS(restricted) do allow for the apparmor profile to not be set (falling back to the default of the container runtime), so we pass validation for both PSS profiles without changes.
Path forward
Unfortunately PSPs don't support "Undefined/nil" as a allowed apparmor profile name. So we would either have to add the default profile to all containers (which is rather complicated to do with out templating as it is a pod level annotation for each container) or we disable PSP validation of apparmor profiles as well during the migration.
Add basic securityContext to all containers T362978
To pass PSS(restricted) all containers will need to explicitly define (at least):
securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true seccompProfile: type: RuntimeDefault
Todos
- Add basic securityContext to all containers: T362978
The following steps need to/can be done on a per cluster basis, controlled by the PodSecurityStandards structure in helmfile.d/admin_ng/values/*/values.yaml:
# Configure the default PodSecurityStandard settings, see: T273507 PodSecurityStandard: disablePSPMutations: true # Disable PSP mutation, allow all seccomp profiles enforce: true # Enforce the PodSecurityStandard profile "restricted" disableRestrictedPSP: true # Disable PSP binding for the restricted PSP
wikikube-staging
- Disable PSP mutation, allow all seccomp profiles (seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*')
- Enforce restricted PSS on all namespaces that use restricted PSP
- Disable the restricted PSP for these namespaces
wikikube (and all others)
- Disable PSP mutation, allow all seccomp profiles (seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*')
- Enforce restricted PSS on all namespaces that use restricted PSP
- Disable the restricted PSP for these namespaces
Migrate all the clusters
- T369491: Migrate aux cluster off of Pod Security Policies
- T369492: Migrate dse cluster off of Pod Security Policies
- T369493: Migrate ml-staging/ml-serve clusters off of Pod Security Policies
Remove related code from admin_ng
Might not be feasible to do right away as wikikube has keep the MediaWiki PSP around until after the k8s upgrade
- Code that adds PSPs
- Code that adds PSP rolebindings