Maniphest T362050

toolforge: review pod templates for PSP replacement
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• aborrero
	Apr 8 2024, 10:12 AM

Description

In both jobs-api and webservices we need to review the pod templates. Previously, the PodSecurityPolicy mechanism would mutate Pod resources to meet the security criteria.

With T279110: [infra] Replace PodSecurityPolicy in Toolforge Kubernetes this will no longer be the case, so we need to natively create Pod resources with a configuration that matches both PSP and the replacement policy, a Kyverno policy. At least during the migration period.

For example, we could make sure our templates include:

securityContext:
  allowPrivilegeEscalation: false
  runAsNonRoot: true
  privileged: false
  hostNetwork: false
  hostIPC: false
  hostPID: false
  capabilities:
    drop:
    - ALL
  seccompProfile:
    type: "runtime/default"

In other words, Pods created both pre-PSP and post-PSP need to validate against the same kyverno policies.

See pod-level securityContext documentation here: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podsecuritycontext-v1-core
See container-level securityContext documentation here: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#securitycontext-v1-core

Details

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
maintain-kubeusers: bump to 0.0.156-20240626103707-3aa9727d	repos/cloud/toolforge/toolforge-deploy!353	ghost	bump_maintain-kubeusers	main
d/changelog: bump to 0.103.9	repos/cloud/toolforge/webservice-cli!47	aborrero	arturo-114-d-changelog-bump-to	main
kyverno_pod_policy: don't autogenerate validation rules	repos/cloud/toolforge/maintain-kubeusers!51	aborrero	arturo-101-kyverno_pod_policy	main
maintain-kubeusers: bump to 0.0.154-20240625155114-8428f7d3	repos/cloud/toolforge/toolforge-deploy!350	ghost	bump_maintain-kubeusers	main
kyverno_pod_policy: validate fsGroup setting only if present	repos/cloud/toolforge/maintain-kubeusers!50	aborrero	arturo-202-kyverno_pod_policy	main
d/changelog: bump to 0.103.8	repos/cloud/toolforge/webservice-cli!44	aborrero	arturo-182-d-changelog-bump-to	main
jobs-api: bump to 0.0.310-20240625090205-108e6a0f	repos/cloud/toolforge/toolforge-deploy!348	ghost	bump_jobs-api	main
jobs: drop ProcMount from securityContext	repos/cloud/toolforge/jobs-api!97	aborrero	arturo-277-jobs-drop-procmount	main
kubernetes: drop ProcMount from securityContext	repos/cloud/toolforge/webservice-cli!43	aborrero	arturo-190-kubernetes-drop-pro	main
functional-tests,webservice: add shell test	repos/cloud/toolforge/toolforge-deploy!347	dcaro	add_shell_test	main
d/changelog: bump to 0.103.7	repos/cloud/toolforge/webservice-cli!42	aborrero	arturo-274-d-changelog-bump-to	main
kubernetes: introduce securityContext in the pod template	repos/cloud/toolforge/webservice-cli!37	aborrero	arturo-104-kubernetes-introduc	main
jobs-api: bump to 0.0.300-20240507123100-0371f944	repos/cloud/toolforge/toolforge-deploy!279	ghost	bump_jobs-api	main
jobs: fix securityContext readOnlyRootFilesystem	repos/cloud/toolforge/jobs-api!83	aborrero	arturo-107-jobs-fix-securityco	main
jobs-api: bump to 0.0.299-20240507120229-eb816a7d	repos/cloud/toolforge/toolforge-deploy!278	ghost	bump_jobs-api	main
jobs-api: introduce securityContext in the pod template	repos/cloud/toolforge/jobs-api!75	aborrero	arturo-58-jobs-api-introduce-s	main

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Raymond_Ndibe	T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
			Restricted Task
Resolved		Slst2020	T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26
Resolved		• aborrero	T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25
Resolved		• aborrero	T279110 [infra] Replace PodSecurityPolicy in Toolforge Kubernetes
Resolved	BUG REPORT	AntiCompositeNumber	T368463 `webservice` (build 0.103.8) crashes on login-buster.toolforge.org (python 3.7)
Resolved		• aborrero	T362050 toolforge: review pod templates for PSP replacement
Resolved		• aborrero	T362966 lima-kilo: replicate sssd setup from Toolforge

Event Timeline

• aborrero created this task.Apr 8 2024, 10:12 AM

readOnlyRootFilesystem: true

We probably don't want to enforce this, so people can create temporary files and similar without the need of mounting volumes

In T362050#9697074, @dcaro wrote:
readOnlyRootFilesystem: true
We probably don't want to enforce this, so people can create temporary files and similar without the need of mounting volumes

I got that one wrong. Yes, indeed the PSP we have today don't enforce this to true, but to false, which is the default anyway https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/

• aborrero updated the task description. (Show Details)Apr 8 2024, 12:08 PM

• aborrero moved this task from Backlog to Next on the User-aborrero board.Apr 11 2024, 9:08 AM

• aborrero changed the task status from Open to In Progress.Apr 17 2024, 9:20 AM

• aborrero triaged this task as Medium priority.

• aborrero moved this task from Next to Doing on the User-aborrero board.

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/75

Draft: jobs-api: introduce securityContext in the pod template

• aborrero changed the status of subtask T362966: lima-kilo: replicate sssd setup from Toolforge from Open to In Progress.Apr 19 2024, 10:10 AM

• aborrero closed subtask T362966: lima-kilo: replicate sssd setup from Toolforge as Resolved.Apr 25 2024, 2:58 PM

dcaro moved this task from Backlog to Toolforge iteration 09 on the Toolforge board.Apr 30 2024, 7:46 AM

dcaro edited projects, added Toolforge (Toolforge iteration 09); removed Toolforge.

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 09) board.

• aborrero mentioned this in T364297: [k8s,infra] track PSP migration plan.May 6 2024, 12:27 PM

• aborrero renamed this task from review pod templates for stricter security to toolforge: review pod templates for PSP replacement.May 6 2024, 12:56 PM

• aborrero updated the task description. (Show Details)

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/75

jobs-api: introduce securityContext in the pod template

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/278

jobs-api: bump to 0.0.299-20240507120229-eb816a7d

TODO: webservice

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/278

jobs-api: bump to 0.0.299-20240507120229-eb816a7d

Before patch https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/278 with only PSP, a Pod resource would have:

at container level:

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL
  runAsGroup: 54005
  runAsUser: 54005

at pod level:

securityContext:
  fsGroup: 54005
  seccompProfile:
    type: RuntimeDefault
  supplementalGroups:
  - 1

With the above patch deployed:

at container-level:

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL
  privileged: false
  procMount: Default
  readOnlyRootFilesystem: true
  runAsGroup: 54005
  runAsNonRoot: true
  runAsUser: 54005

at pod level:

securityContext:
  fsGroup: 54005
  runAsGroup: 54005
  runAsNonRoot: true
  runAsUser: 54005
  seccompProfile:
    type: RuntimeDefault
  supplementalGroups:
  - 1

The supplementalGroups is added by PSP, not by the the jobs-api template.

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/83

jobs: fix securityContext readOnlyRootFilesystem

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/83

jobs: fix securityContext readOnlyRootFilesystem

Maintenance_bot removed a project: Patch-For-Review.May 7 2024, 12:31 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/279

jobs-api: bump to 0.0.300-20240507123100-0371f944

• aborrero updated the task description. (Show Details)May 7 2024, 12:37 PM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/279

jobs-api: bump to 0.0.300-20240507123100-0371f944

Maintenance_bot removed a project: Patch-For-Review.May 7 2024, 1:31 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/37

kubernetes: introduce securityContext in the pod template

• aborrero moved this task from Doing to Next on the User-aborrero board.May 22 2024, 1:42 PM

dcaro edited projects, added Toolforge (Toolforge iteration 10); removed Toolforge (Toolforge iteration 09).May 22 2024, 4:40 PM

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 10) board.

dcaro edited projects, added Toolforge (Toolforge iteration 11); removed Toolforge (Toolforge iteration 10).Jun 4 2024, 2:56 PM

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 11) board.

• aborrero moved this task from Next to Doing on the User-aborrero board.Jun 7 2024, 3:46 PM

• aborrero moved this task from Doing to Next on the User-aborrero board.Jun 20 2024, 9:23 AM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/37

kubernetes: introduce securityContext in the pod template

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2024, 12:30 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/42

d/changelog: bump to 0.103.7

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/42

d/changelog: bump to 0.103.7

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2024, 1:31 PM

Mentioned in SAL (#wikimedia-cloud) [2024-06-24T15:44:57Z] <arturo> deploy toolforge-webservice 0.103.7 (T362050)

Mentioned in SAL (#wikimedia-cloud) [2024-06-24T15:45:01Z] <arturo> deploy toolforge-webservice 0.103.7 (T362050)

$ webservice perl5.36 shell --mount=all
Error from server (Forbidden): pods "shell-1719270590" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.procMount: Invalid value: "DefaultProcMount": ProcMountType is not allowed]
$ webservice php8.2 shell --mount=all
Error from server (Forbidden): pods "shell-1719270759" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.procMount: Invalid value: "DefaultProcMount": ProcMountType is not allowed]
$ webservice python3.11 shell --mount=all
Error from server (Forbidden): pods "shell-1719270775" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.procMount: Invalid value: "DefaultProcMount": ProcMountType is not allowed]
$ dpkg -l | grep toolforge-webservice
ii  toolforge-webservice                 0.103.7                              all          Infrastructure for running webservices on Toolforge

[23:14]  <   anomie> Looks like it still works from login-buster.toolforge.org 🤷

$ ssh login-buster.toolforge.org
$ $ dpkg -l | grep toolforge-webservice
ii  toolforge-webservice                                        0.103.6                            all          Infrastructure for running webservices on Toolforge

Until webservice shell is fixed generally, hacky workarounds are:

Run from login-buster.toolforge.org which has an older version of webservice
Manually start an interactive container in the style documented at https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Launch_an_interactive_shell_in_the_cluster

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/43

kubernetes: drop ProcMount from securityContext

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/97

jobs: drop ProcMount from securityContext

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/43

kubernetes: drop ProcMount from securityContext

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/97

jobs: drop ProcMount from securityContext

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/348

jobs-api: bump to 0.0.310-20240625090205-108e6a0f

In T362050#9919714, @bd808 wrote:

$ webservice perl5.36 shell --mount=all
Error from server (Forbidden): pods "shell-1719270590" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.procMount: Invalid value: "DefaultProcMount": ProcMountType is not allowed]
$ webservice php8.2 shell --mount=all
Error from server (Forbidden): pods "shell-1719270759" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.procMount: Invalid value: "DefaultProcMount": ProcMountType is not allowed]
$ webservice python3.11 shell --mount=all
Error from server (Forbidden): pods "shell-1719270775" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.procMount: Invalid value: "DefaultProcMount": ProcMountType is not allowed]
$ dpkg -l | grep toolforge-webservice
ii  toolforge-webservice                 0.103.7                              all          Infrastructure for running webservices on Toolforge

This turned out to be an interesting bug, in several stages:

the right value for procMount is Default, as documented in the kubernetes source code itself: https://github.com/kubernetes/api/blob/release-1.24/core/v1/types.go#L6422
the upstream API docs for the securityContext contains a typo, and refers to the default value as DefaultProcMount. See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#securitycontext-v1-core
the upstream docs for PodSecurityPolicy also contains the typo. See https://web.archive.org/web/20240325013003/https://v1-24.docs.kubernetes.io/docs/concepts/security/pod-security-policy/#allowedprocmounttypes
I followed the upstream API docs with the typo when introducing the securityContext into both jobs-api and webservice (the original changes in this very ticket and related patches). Therefore, I introduced the wrong value following the upstream docs.
The bug did not immediately trigger though, because we were testing toolforge jobs and webservices, which generate pods via Deployments and other generators, via a pod template. Kubernetes ignores the wrong value in the pod template when generating a new pod. Pods generated would use Default, the right value, and therefore the bug would not trigger
However, webservice shell creates a pod directly, without an intermediate Deployment or other pod generator. Because of this, the wrong procMount value would be used directly, and the API would rightfully complain.