Page MenuHomePhabricator
Create Task
Maniphest T265557

[tbs.maintainkubeusers] Create security policies for running buildpack images
Closed, InvalidPublic
Actions

Description

The cloud-native buildpack specification calls for a statically-defined UID and GID for a builder container. That means that unless we like the notion of exponentially increasing docker image disk space usage, we need to standardize a single UID/GID for buildpack-style containers, which we've decided will be 61312.

We've discovered that a pod security policy can be created that allows that UID but does not allow the primary NFS mount for Toolforge, which prevents conflicts with a tool's permissions on NFS. Let's teach maintain-kubeusers how to create PSP assignments for this and backfill it to existing users to allow the usage of these containers on Toolforge Kubernetes.

The PSP required can be shared among all tools since the values are not different per-tool. The roles and bindings need to be created by maintain-kubeusers because those are namespaced (unless this service ends up in a shared namespace--which seems unlikely because of quotas).

TO BE REFINED (PSPs will be removed in k8s 1.25)

Details

SubjectRepoBranchLines +/-
toolforge k8s: add a PodSecurityPolicy to be used by buildpacksoperations/puppetproduction+57 -0
Add PSP roles for buildpack imageslabs/tools/maintain-kubeusersmaster+547 -296
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Bstorm triaged this task as Medium priority.Oct 14 2020, 11:18 PM
• Bstorm created this task.
• Bstorm moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.
• Bstorm updated the task description. (Show Details)
• Bstorm added a comment.Oct 15 2020, 6:36 PM

For reference, this yaml worked correctly in manual testing in toolsbeta:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
  name: cnb-tool-test-psp
spec:
  allowPrivilegeEscalation: false
  allowedHostPaths:
  - pathPrefix: /public/dumps
    readOnly: true
  - pathPrefix: /mnt/nfs
    readOnly: true
  - pathPrefix: /etc/wmcs-project
    readOnly: true
  - pathPrefix: /etc/ldap.yaml
    readOnly: true
  - pathPrefix: /etc/novaobserver.yaml
    readOnly: true
  - pathPrefix: /etc/ldap.conf
    readOnly: true
  fsGroup:
    ranges:
    - max: 1000
      min: 1000
    rule: MustRunAs
  requiredDropCapabilities:
  - ALL
  runAsGroup:
    ranges:
    - max: 1000
      min: 1000
    rule: MustRunAs
  runAsUser:
    ranges:
    - max: 1000
      min: 1000
    rule: MustRunAs
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - projected
  - secret
  - hostPath
  - persistentVolumeClaim
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cnb-tool-test-psp
  namespace: tool-test
rules:
- apiGroups:
  - extensions
  resourceNames:
  - cnb-tool-test-psp
  resources:
  - podsecuritypolicies
  verbs:
  - use
---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cnb-tool-test-psp-binding
  namespace: tool-test
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cnb-tool-test-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: test
• Bstorm updated the task description. (Show Details)Oct 15 2020, 6:38 PM
Legoktm added a parent task: T265681: [tbs.tools] Deploy a tool using a buildpack-based image on Toolforge.Oct 15 2020, 9:22 PM
Legoktm updated the task description. (Show Details)Oct 15 2020, 9:38 PM
gerritbot added a comment.Oct 21 2020, 12:02 AM

Change 635403 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[labs/tools/maintain-kubeusers@master] [WIP] Add PSP for buildpack images

https://gerrit.wikimedia.org/r/635403

gerritbot added a project: Patch-For-Review.Oct 21 2020, 12:02 AM
gerritbot added a comment.Oct 21 2020, 8:49 PM

Change 635641 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge k8s: add a PodSecurityPolicy to be used by buildpacks

https://gerrit.wikimedia.org/r/635641

gerritbot added a comment.Oct 22 2020, 4:08 PM

Change 635641 merged by Bstorm:
[operations/puppet@production] toolforge k8s: add a PodSecurityPolicy to be used by buildpacks

https://gerrit.wikimedia.org/r/635641

Stashbot added a comment.Oct 22 2020, 4:22 PM

Mentioned in SAL (#wikimedia-cloud) [2020-10-22T16:22:32Z] <bstorm> created buildpack psp for T265557

• Bstorm added a comment.Oct 22 2020, 4:23 PM
root@toolsbeta-test-k8s-control-1:~# kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml
clusterrole.rbac.authorization.k8s.io/tool-observer unchanged
podsecuritypolicy.policy/toolforge-tfb-psp created
gerritbot added a comment.Oct 23 2020, 12:05 AM

Change 635403 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] Add PSP roles for buildpack images

https://gerrit.wikimedia.org/r/635403

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2020, 12:10 AM
• Bstorm added a comment.Oct 23 2020, 12:34 AM

Ran this in toolsbeta with the new code:

root@toolsbeta-test-k8s-control-1:/srv/git/maintain-kubeusers# kubectl -n maintain-kubeusers exec -it maintain-kubeusers-ops -- /bin/ash
/app # source venv/bin/activate
(venv) /app # python maintain_kubeusers.py --force-buildpack-psp --once
starting a run

And then it looks like it created the stuff:

root@toolsbeta-test-k8s-control-1:/srv/git/maintain-kubeusers# kubectl -n tool-test get roles
NAME            AGE
tfb-test-psp    64s
tool-test-psp   351d

root@toolsbeta-test-k8s-control-1:/srv/git/maintain-kubeusers# kubectl -n tool-test get rolebinding
NAME                       AGE
default-test-psp-binding   351d
test-obs-psp               309d
test-tool-binding          351d
tfb-test-psp-binding       112s
tool-test-psp-binding      351d
Legoktm moved this task from Soon! to Doing on the cloud-services-team (Kanban) board.Oct 23 2020, 10:30 PM
Andrew mentioned this in T265681: [tbs.tools] Deploy a tool using a buildpack-based image on Toolforge.Feb 16 2021, 7:20 PM
aborrero assigned this task to • Bstorm.May 11 2021, 4:41 PM
dcaro added a project: User-dcaro.Oct 18 2021, 1:36 PM
dcaro edited projects, added Toolforge Build Service; removed Toolforge.Oct 18 2021, 2:10 PM
dcaro added a project: Cloud-Services-Origin-Team.Nov 24 2021, 3:25 PM
dcaro added a project: Cloud-Services-Worktype-Project.
dcaro claimed this task.Dec 7 2021, 8:38 AM
dcaro moved this task from To refine to Refined on the User-dcaro board.
dcaro moved this task from Refined to To refine on the User-dcaro board.Dec 7 2021, 8:41 AM
dcaro moved this task from To refine to Refined on the User-dcaro board.Apr 14 2022, 8:37 AM
dcaro renamed this task from Modify maintain-kubeusers to allow for buildpacks to [tbs.harbor] Create harbor namespaces using maintainkubeusers script.Aug 26 2022, 7:46 AM
dcaro renamed this task from [tbs.harbor] Create harbor namespaces using maintainkubeusers script to [tbs.maintainkubeusers] Create security policies for running buildpack images.
dcaro updated the task description. (Show Details)
dcaro removed a subscriber: • Bstorm.
dcaro removed dcaro as the assignee of this task.Aug 26 2022, 8:52 AM
dcaro removed a parent task: T194332: [Epic,builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs.
dcaro subscribed.
dcaro added a parent task: T316330: [tbs.toolsbeta] Deploy a tool using a buildpack-based image on Toolsbeta.Aug 26 2022, 9:44 AM
dcaro removed a parent task: T265681: [tbs.tools] Deploy a tool using a buildpack-based image on Toolforge.
dcaro closed this task as Invalid.Nov 7 2022, 1:29 PM

this is not needed anymore as we are not running the builders on the user's namespaces.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL