Page MenuHomePhabricator

Design the resource limits, RBAC and PSP needed for the PAWS Kubernetes cluster
Closed, ResolvedPublic

Description

To improve the security and resiliency for PAWS, it's new cluster should have some measure of PSP and RBAC protection. Since the setup is largely going to be a static thing, it seems reasonable to deploy a kustomize directory that can be hooked to kubectl on refresh in puppet. Emergency changes will be possible by admins, but puppet will update whenever puppet is changed. That should be a fair balance.

Event Timeline

Bstorm triaged this task as Medium priority.Apr 28 2020, 4:37 PM
Bstorm created this task.

Change 589454 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] admins: Introduce admin user creation for projects

https://gerrit.wikimedia.org/r/589454

Change 589454 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] admins: Introduce admin user creation for projects

https://gerrit.wikimedia.org/r/589454

Change 596763 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] admins: Fixed critical typo

https://gerrit.wikimedia.org/r/596763

Change 596763 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] admins: Fixed critical typo

https://gerrit.wikimedia.org/r/596763

Change 596770 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] admins: Another bugfix -- duplicate names of clusterrolebindings

https://gerrit.wikimedia.org/r/596770

Change 596770 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] admins: Another bugfix -- duplicate names of clusterrolebindings

https://gerrit.wikimedia.org/r/596770

Change 596781 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] admins: mount the NFS home dirs in the container

https://gerrit.wikimedia.org/r/596781

Change 596781 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] admins: mount the NFS home dirs in the container

https://gerrit.wikimedia.org/r/596781

At this point, we should be able to deploy maintain-kubeusers with a nice deployment already set in it for paws to have admin users (the paws project will just need a paws.admin LDAP group to key off of).

Docs are here for how they work https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Admin_accounts

The only other thing paws needs is RBAC and PSP for the containers that will need to run, which requires the right RBAC for whatever is deploying things and the service account the pods run as (and a PSP to constrain all that as well). Since right now it uses helm2, there's a bit of re-engineering needed.

This should prevent admins from ever getting locked out by an expiring cert, btw.

So Jupyterhub's chart will handle some of its own RBAC. That said, the deployhook needs some.
https://github.com/crookedstorm/paws/blob/master/paws/templates/deploy-hook.yaml This is my proposal for now. It needs to be able to create serviceaccounts, bindings and give at least one serviceaccount access to the kube-scheduler at the cluster scope.

helm3 will just get the credentials like kubectl of the serviceaccount of the pod it is running as.

We need some PSP for this now!

We need some PSP for this now!

I think that can just go in the profile for control-plane servers. A forked version of PAWS wouldn't need it without our puppetization.

Change 598863 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] contexts: context should be correct for project

https://gerrit.wikimedia.org/r/598863

Change 598863 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] contexts: context should be correct for project

https://gerrit.wikimedia.org/r/598863

Bstorm renamed this task from Design the RBAC and PSP needed for the PAWS Kubernetes cluster to Design the resource limits, RBAC and PSP needed for the PAWS Kubernetes cluster.Jun 17 2020, 9:26 PM

The helm chart for jupyterhub uses the verbiage "guarantee" and "limit" for resource limits on user pods. By default, there is no resource limits or guarantee for CPU, which is pretty bad, and there is a guarantee of 1GB (which makes sense for python).

I'll add these to the latest helm chart:

memory:
  guarantee: 1G
  limit: 3G
cpu:
  guarantee: .5
  limit: 1

To be clear: guarantee will affect scheduling while the limit will stop a kernel dead in its tracks if it goes over it. It's a fail-safe to stop a single user from taking down a node. So basically, the scheduler will pack in up to 7 user pods per node (8GB with resource guarantees for daemonsets) and each one can peak out at 3GB for short periods. For CPU, it would pack no more than 8 in as well (4 CPU per node), but each can claim no more than 1 entire CPU at a time.

Mentioned in SAL (#wikimedia-cloud) [2020-06-17T21:51:48Z] <bstorm_> upgraded chart in the new cluster to include resource limits T251298

Makes sense to me. (the terms are from jupyterhub where limits and guarantees were initially backed by other components)

Makes sense to me. (the terms are from jupyterhub where limits and guarantees were initially backed by other components)

Fair enough!

Now that OAUTH is up, I am able to prove there is definitely a failure.

[E 2020-06-23 22:54:13.719 JupyterHub pages:325] Previous spawn for BStorm_(WMF) failed: (403)
    Reason: error
    HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 23 Jun 2020 22:54:13 GMT', 'Content-Length': '993'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"jupyter--42-53torm-5f-28-57-4d-46-29\" is forbidden: unable to validate against any pod security policy: [spec.containers[0].volumeMounts[1].readOnly: Invalid value: false: must be read-only spec.containers[0].volumeMounts[2].readOnly: Invalid value: false: must be read-only spec.containers[0].volumeMounts[3].readOnly: Invalid value: false: must be read-only spec.initContainers[0].securityContext.runAsUser: Invalid value: 0: must be in the ranges: [{52771 52771}] spec.initContainers[1].securityContext.runAsUser: Invalid value: 0: must be in the ranges: [{52771 52771}] spec.initContainers[1].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.initContainers[1].securityContext.capabilities.add: Invalid value: \"NET_ADMIN\": capability may not be added]","reason":"Forbidden","details":{"name":"jupyter--42-53torm-5f-28-57-4d-46-29","kind":"pods"},"code":403

So It is not setting the runAsUser correctly somewhere, needs readOnly set in a place I'm sure of where and the PSP needs a couple things.

Ok, so! I'm kicking things back and forth and ended up working locally because some things are not obvious.

Jupyterhub runs a privileged initcontainer even with initcontainers blanked to do this:

'init_containers': [{'args': None,
                     'command': ['iptables',
                                 '-A',
                                 'OUTPUT',
                                 '-d',
                                 '169.254.169.254',
                                 '-j',
                                 'DROP'],
                     'env': None,
                     'env_from': None,
                     'image': 'jupyterhub/k8s-network-tools:0.9.0',
                     'image_pull_policy': None,
                     'lifecycle': None,
                     'liveness_probe': None,
                     'name': 'block-cloud-metadata',
                     'ports': None,
                     'readiness_probe': None,
                     'resources': None,
                     'security_context': {'allow_privilege_escalation': None,
                                          'capabilities': {'add': ['NET_ADMIN'],
                                                           'drop': None},
                                          'privileged': True,
                                          'proc_mount': None,
                                          'read_only_root_filesystem': None,
                                          'run_as_group': None,
                                          'run_as_non_root': None,
                                          'run_as_user': 0,
                                          'se_linux_options': None},
                     'stdin': None,
                     'stdin_once': None,
                     'termination_message_path': None,
                     'termination_message_policy': None,
                     'tty': None,
                     'volume_devices': None,
                     'volume_mounts': None,
                     'working_dir': None}],

That is blocked by PSP, of course. Luckily, it is configurable! https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/fcf9c2e92a450e175a9d53abbe6dbbb2936df096/jupyterhub/files/hub/jupyterhub_config.py#L430


PROGRESS!

Ok, so now it is complaining about /home/paws, which it has a root initcontainer to chown. I'm looking for a workaround for that.

Ok, so I am still getting PermissionError: [Errno 13] Permission denied: '/home/paws/.local' because SOMEHOW it creates the userhome as root.
I am not 100% sure how on earth that is possible because everything is running as the paws user, but there it is. I may need to set the initcontainer to run as a privileged setup somehow, but I'm really hoping not to.

Got it! The problem is letting docker create the dirs. I simply added an explicit creation as tools.paws by a container instead.

However, we have a redirecting problem once the server is up. I get endless 302s and Firefox craps out.

The redirect is in the singleuser pod:

bstorm@paws-k8s-control-1:~/src/paws$ kubectl -n prod logs jupyter--42-53torm-5f-28-57-4d-46-29
[I 2020-06-24 01:50:39.889 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /srv/paws/lib/python3.6/site-packages/jupyterlab
[I 2020-06-24 01:50:39.889 SingleUserNotebookApp extension:159] JupyterLab application directory is /srv/paws/share/jupyter/lab
[I 2020-06-24 01:50:39.926 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[I 2020-06-24 01:50:39.934 SingleUserNotebookApp notebookapp:1924] Serving notebooks from local directory: /home/paws
[I 2020-06-24 01:50:39.934 SingleUserNotebookApp notebookapp:1924] The Jupyter Notebook is running at:
[I 2020-06-24 01:50:39.934 SingleUserNotebookApp notebookapp:1924] http://jupyter--42-53torm-5f-28-57-4d-46-29:8888/user/BStorm_%28WMF%29/
[I 2020-06-24 01:50:39.934 SingleUserNotebookApp notebookapp:1925] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2020-06-24 01:50:39.952 SingleUserNotebookApp singleuser:542] Updating Hub with activity every 300 seconds
[I 2020-06-24 01:50:42.143 SingleUserNotebookApp log:174] 302 GET /user/BStorm_%28WMF%29/ -> /user/BStorm_%28WMF%29/tree? (@192.168.20.180) 1.62ms

That redirect seems correct, as that's what happens in the existing paws, but somehow it tries to redirect back to hub, possibly in the ingress layer for T195217: Simplify ingress methods for PAWS FYI @aborrero

I'll dig around and see.

Yup, it looks like that redirect from /user back to /hub is at the ingress and is not quite right. It needs to be able to hit user pods as well. Yay! PSP works. I also removed 2 root actions by the jhub.

Looks like the ingress objects that exist are created by helm and are...old:

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/tls-acme: "true"
      meta.helm.sh/release-name: paws
      meta.helm.sh/release-namespace: prod
    creationTimestamp: "2020-06-23T23:55:42Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
    name: deploy-hook
    namespace: prod
    resourceVersion: "6173412"
    selfLink: /apis/extensions/v1beta1/namespaces/prod/ingresses/deploy-hook
    uid: d275afde-5ccc-4436-980f-248baa073f5e
  spec:
    rules:
    - host: paws-deploy-hook.tools.wmflabs.org
      http:
        paths:
        - backend:
            serviceName: deploy-hook
            servicePort: 8888
          path: /
    tls:
    - hosts:
      - paws-deploy-hook.tools.wmflabs.org
      secretName: kubelego-deploy-hook-paws
  status:
    loadBalancer: {}
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"hub","namespace":"prod"},"spec":{"rules":[{"host":"paws.wmcloud.org","http":{"paths":[{"backend":{"serviceName":"hub","servicePort":8081}}]}},{"host":"paws.wmflabs.org","http":{"paths":[{"backend":{"serviceName":"hub","servicePort":8081}}]}}]}}
    creationTimestamp: "2020-06-19T11:24:50Z"
    generation: 1
    name: hub
    namespace: prod
    resourceVersion: "5008699"
    selfLink: /apis/extensions/v1beta1/namespaces/prod/ingresses/hub
    uid: 6c1fed46-acdf-457e-830f-928980501429
  spec:
    rules:
    - host: paws.wmcloud.org
      http:
        paths:
        - backend:
            serviceName: hub
            servicePort: 8081
    - host: paws.wmflabs.org
      http:
        paths:
        - backend:
            serviceName: hub
            servicePort: 8081
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

That all needs a bit of love one way or another.

@aborrero I fixed it! I had to delete the ingress object named hub, which may have been manually created. Then I had to re-enable the ingress in jupyterhub.

It's name is jupyterhub. We have a working prototype!!!!

Bstorm closed this task as Resolved.Jun 24 2020, 5:24 PM

To be crystal clear, I am unblocked on paws.

Bstorm reopened this task as Open.Jun 24 2020, 6:02 PM

I'm actually going to re-open this because helm still doesn't have a user impersonation option. That means that I should make maintain-kubeusers set admins to "edit" instead of "view" by default in PAWS, leaving the impersonation RBAC in place.

Bstorm closed this task as Resolved.Jun 24 2020, 6:03 PM

We can agree to just leave manual deploys to root as well. That's ok. kubectl can impersonate cluster-admin just fine for admins. Actually, I think that is ok as long as we stick to the upgrade schedules of Toolforge

That and apparently I just got one of the issues for adding --as to helm reopened https://github.com/helm/helm/issues/5303