Page MenuHomePhabricator

Modify maintain-kubeusers to allow for buildpacks
Open, MediumPublic

Description

The cloud-native buildpack specification calls for a statically-defined UID and GID for a builder container. That means that unless we like the notion of exponentially increasing docker image disk space usage, we need to standardize a single UID/GID for buildpack-style containers, which we've decided will be 61312.

We've discovered that a pod security policy can be created that allows that UID but does not allow the primary NFS mount for Toolforge, which prevents conflicts with a tool's permissions on NFS. Let's teach maintain-kubeusers how to create PSP assignments for this and backfill it to existing users to allow the usage of these containers on Toolforge Kubernetes.

The PSP required can be shared among all tools since the values are not different per-tool. The roles and bindings need to be created by maintain-kubeusers because those are namespaced (unless this service ends up in a shared namespace--which seems unlikely because of quotas).

Event Timeline

Bstorm triaged this task as Medium priority.Oct 14 2020, 11:18 PM
Bstorm created this task.
Bstorm moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.
Bstorm updated the task description. (Show Details)

For reference, this yaml worked correctly in manual testing in toolsbeta:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
  name: cnb-tool-test-psp
spec:
  allowPrivilegeEscalation: false
  allowedHostPaths:
  - pathPrefix: /public/dumps
    readOnly: true
  - pathPrefix: /mnt/nfs
    readOnly: true
  - pathPrefix: /etc/wmcs-project
    readOnly: true
  - pathPrefix: /etc/ldap.yaml
    readOnly: true
  - pathPrefix: /etc/novaobserver.yaml
    readOnly: true
  - pathPrefix: /etc/ldap.conf
    readOnly: true
  fsGroup:
    ranges:
    - max: 1000
      min: 1000
    rule: MustRunAs
  requiredDropCapabilities:
  - ALL
  runAsGroup:
    ranges:
    - max: 1000
      min: 1000
    rule: MustRunAs
  runAsUser:
    ranges:
    - max: 1000
      min: 1000
    rule: MustRunAs
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - projected
  - secret
  - hostPath
  - persistentVolumeClaim
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cnb-tool-test-psp
  namespace: tool-test
rules:
- apiGroups:
  - extensions
  resourceNames:
  - cnb-tool-test-psp
  resources:
  - podsecuritypolicies
  verbs:
  - use
---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cnb-tool-test-psp-binding
  namespace: tool-test
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cnb-tool-test-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: test

Change 635403 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[labs/tools/maintain-kubeusers@master] [WIP] Add PSP for buildpack images

https://gerrit.wikimedia.org/r/635403

Change 635641 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge k8s: add a PodSecurityPolicy to be used by buildpacks

https://gerrit.wikimedia.org/r/635641

Change 635641 merged by Bstorm:
[operations/puppet@production] toolforge k8s: add a PodSecurityPolicy to be used by buildpacks

https://gerrit.wikimedia.org/r/635641

Mentioned in SAL (#wikimedia-cloud) [2020-10-22T16:22:32Z] <bstorm> created buildpack psp for T265557

root@toolsbeta-test-k8s-control-1:~# kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml
clusterrole.rbac.authorization.k8s.io/tool-observer unchanged
podsecuritypolicy.policy/toolforge-tfb-psp created

Change 635403 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] Add PSP roles for buildpack images

https://gerrit.wikimedia.org/r/635403

Ran this in toolsbeta with the new code:

root@toolsbeta-test-k8s-control-1:/srv/git/maintain-kubeusers# kubectl -n maintain-kubeusers exec -it maintain-kubeusers-ops -- /bin/ash
/app # source venv/bin/activate
(venv) /app # python maintain_kubeusers.py --force-buildpack-psp --once
starting a run

And then it looks like it created the stuff:

root@toolsbeta-test-k8s-control-1:/srv/git/maintain-kubeusers# kubectl -n tool-test get roles
NAME            AGE
tfb-test-psp    64s
tool-test-psp   351d

root@toolsbeta-test-k8s-control-1:/srv/git/maintain-kubeusers# kubectl -n tool-test get rolebinding
NAME                       AGE
default-test-psp-binding   351d
test-obs-psp               309d
test-tool-binding          351d
tfb-test-psp-binding       112s
tool-test-psp-binding      351d