To improve the security and resiliency for PAWS, it's new cluster should have some measure of PSP and RBAC protection. Since the setup is largely going to be a static thing, it seems reasonable to deploy a kustomize directory that can be hooked to kubectl on refresh in puppet. Emergency changes will be possible by admins, but puppet will update whenever puppet is changed. That should be a fair balance.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Restricted Task | |||||
Resolved | Bstorm | T246122 Upgrade the Toolforge Kubernetes cluster to v1.16 | |||
Resolved | Bstorm | T211096 PAWS: Rebuild and upgrade Kubernetes | |||
Resolved | Bstorm | T251298 Design the resource limits, RBAC and PSP needed for the PAWS Kubernetes cluster |
Event Timeline
Change 589454 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] admins: Introduce admin user creation for projects
Change 589454 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] admins: Introduce admin user creation for projects
Change 596763 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] admins: Fixed critical typo
Change 596763 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] admins: Fixed critical typo
Change 596770 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] admins: Another bugfix -- duplicate names of clusterrolebindings
Change 596770 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] admins: Another bugfix -- duplicate names of clusterrolebindings
Change 596781 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] admins: mount the NFS home dirs in the container
Change 596781 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] admins: mount the NFS home dirs in the container
At this point, we should be able to deploy maintain-kubeusers with a nice deployment already set in it for paws to have admin users (the paws project will just need a paws.admin LDAP group to key off of).
Docs are here for how they work https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Admin_accounts
The only other thing paws needs is RBAC and PSP for the containers that will need to run, which requires the right RBAC for whatever is deploying things and the service account the pods run as (and a PSP to constrain all that as well). Since right now it uses helm2, there's a bit of re-engineering needed.
So Jupyterhub's chart will handle some of its own RBAC. That said, the deployhook needs some.
https://github.com/crookedstorm/paws/blob/master/paws/templates/deploy-hook.yaml This is my proposal for now. It needs to be able to create serviceaccounts, bindings and give at least one serviceaccount access to the kube-scheduler at the cluster scope.
helm3 will just get the credentials like kubectl of the serviceaccount of the pod it is running as.
We need some PSP for this now!
I think that can just go in the profile for control-plane servers. A forked version of PAWS wouldn't need it without our puppetization.
Change 598863 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] contexts: context should be correct for project
Change 598863 merged by jenkins-bot:
[labs/tools/maintain-kubeusers@master] contexts: context should be correct for project
The helm chart for jupyterhub uses the verbiage "guarantee" and "limit" for resource limits on user pods. By default, there is no resource limits or guarantee for CPU, which is pretty bad, and there is a guarantee of 1GB (which makes sense for python).
I'll add these to the latest helm chart:
memory: guarantee: 1G limit: 3G cpu: guarantee: .5 limit: 1
To be clear: guarantee will affect scheduling while the limit will stop a kernel dead in its tracks if it goes over it. It's a fail-safe to stop a single user from taking down a node. So basically, the scheduler will pack in up to 7 user pods per node (8GB with resource guarantees for daemonsets) and each one can peak out at 3GB for short periods. For CPU, it would pack no more than 8 in as well (4 CPU per node), but each can claim no more than 1 entire CPU at a time.
Mentioned in SAL (#wikimedia-cloud) [2020-06-17T21:51:48Z] <bstorm_> upgraded chart in the new cluster to include resource limits T251298
Makes sense to me. (the terms are from jupyterhub where limits and guarantees were initially backed by other components)
Now that OAUTH is up, I am able to prove there is definitely a failure.
[E 2020-06-23 22:54:13.719 JupyterHub pages:325] Previous spawn for BStorm_(WMF) failed: (403) Reason: error HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 23 Jun 2020 22:54:13 GMT', 'Content-Length': '993'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"jupyter--42-53torm-5f-28-57-4d-46-29\" is forbidden: unable to validate against any pod security policy: [spec.containers[0].volumeMounts[1].readOnly: Invalid value: false: must be read-only spec.containers[0].volumeMounts[2].readOnly: Invalid value: false: must be read-only spec.containers[0].volumeMounts[3].readOnly: Invalid value: false: must be read-only spec.initContainers[0].securityContext.runAsUser: Invalid value: 0: must be in the ranges: [{52771 52771}] spec.initContainers[1].securityContext.runAsUser: Invalid value: 0: must be in the ranges: [{52771 52771}] spec.initContainers[1].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.initContainers[1].securityContext.capabilities.add: Invalid value: \"NET_ADMIN\": capability may not be added]","reason":"Forbidden","details":{"name":"jupyter--42-53torm-5f-28-57-4d-46-29","kind":"pods"},"code":403
So It is not setting the runAsUser correctly somewhere, needs readOnly set in a place I'm sure of where and the PSP needs a couple things.
Ok, so! I'm kicking things back and forth and ended up working locally because some things are not obvious.
Jupyterhub runs a privileged initcontainer even with initcontainers blanked to do this:
'init_containers': [{'args': None, 'command': ['iptables', '-A', 'OUTPUT', '-d', '169.254.169.254', '-j', 'DROP'], 'env': None, 'env_from': None, 'image': 'jupyterhub/k8s-network-tools:0.9.0', 'image_pull_policy': None, 'lifecycle': None, 'liveness_probe': None, 'name': 'block-cloud-metadata', 'ports': None, 'readiness_probe': None, 'resources': None, 'security_context': {'allow_privilege_escalation': None, 'capabilities': {'add': ['NET_ADMIN'], 'drop': None}, 'privileged': True, 'proc_mount': None, 'read_only_root_filesystem': None, 'run_as_group': None, 'run_as_non_root': None, 'run_as_user': 0, 'se_linux_options': None}, 'stdin': None, 'stdin_once': None, 'termination_message_path': None, 'termination_message_policy': None, 'tty': None, 'volume_devices': None, 'volume_mounts': None, 'working_dir': None}],
That is blocked by PSP, of course. Luckily, it is configurable! https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/fcf9c2e92a450e175a9d53abbe6dbbb2936df096/jupyterhub/files/hub/jupyterhub_config.py#L430
PROGRESS!
Ok, so now it is complaining about /home/paws, which it has a root initcontainer to chown. I'm looking for a workaround for that.
Ok, so I am still getting PermissionError: [Errno 13] Permission denied: '/home/paws/.local' because SOMEHOW it creates the userhome as root.
I am not 100% sure how on earth that is possible because everything is running as the paws user, but there it is. I may need to set the initcontainer to run as a privileged setup somehow, but I'm really hoping not to.
Got it! The problem is letting docker create the dirs. I simply added an explicit creation as tools.paws by a container instead.
However, we have a redirecting problem once the server is up. I get endless 302s and Firefox craps out.
The redirect is in the singleuser pod:
bstorm@paws-k8s-control-1:~/src/paws$ kubectl -n prod logs jupyter--42-53torm-5f-28-57-4d-46-29 [I 2020-06-24 01:50:39.889 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /srv/paws/lib/python3.6/site-packages/jupyterlab [I 2020-06-24 01:50:39.889 SingleUserNotebookApp extension:159] JupyterLab application directory is /srv/paws/share/jupyter/lab [I 2020-06-24 01:50:39.926 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0 [I 2020-06-24 01:50:39.934 SingleUserNotebookApp notebookapp:1924] Serving notebooks from local directory: /home/paws [I 2020-06-24 01:50:39.934 SingleUserNotebookApp notebookapp:1924] The Jupyter Notebook is running at: [I 2020-06-24 01:50:39.934 SingleUserNotebookApp notebookapp:1924] http://jupyter--42-53torm-5f-28-57-4d-46-29:8888/user/BStorm_%28WMF%29/ [I 2020-06-24 01:50:39.934 SingleUserNotebookApp notebookapp:1925] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [I 2020-06-24 01:50:39.952 SingleUserNotebookApp singleuser:542] Updating Hub with activity every 300 seconds [I 2020-06-24 01:50:42.143 SingleUserNotebookApp log:174] 302 GET /user/BStorm_%28WMF%29/ -> /user/BStorm_%28WMF%29/tree? (@192.168.20.180) 1.62ms
That redirect seems correct, as that's what happens in the existing paws, but somehow it tries to redirect back to hub, possibly in the ingress layer for T195217: Simplify ingress methods for PAWS FYI @aborrero
I'll dig around and see.
Yup, it looks like that redirect from /user back to /hub is at the ingress and is not quite right. It needs to be able to hit user pods as well. Yay! PSP works. I also removed 2 root actions by the jhub.
Looks like the ingress objects that exist are created by helm and are...old:
apiVersion: v1 items: - apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/tls-acme: "true" meta.helm.sh/release-name: paws meta.helm.sh/release-namespace: prod creationTimestamp: "2020-06-23T23:55:42Z" generation: 1 labels: app.kubernetes.io/managed-by: Helm name: deploy-hook namespace: prod resourceVersion: "6173412" selfLink: /apis/extensions/v1beta1/namespaces/prod/ingresses/deploy-hook uid: d275afde-5ccc-4436-980f-248baa073f5e spec: rules: - host: paws-deploy-hook.tools.wmflabs.org http: paths: - backend: serviceName: deploy-hook servicePort: 8888 path: / tls: - hosts: - paws-deploy-hook.tools.wmflabs.org secretName: kubelego-deploy-hook-paws status: loadBalancer: {} - apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"hub","namespace":"prod"},"spec":{"rules":[{"host":"paws.wmcloud.org","http":{"paths":[{"backend":{"serviceName":"hub","servicePort":8081}}]}},{"host":"paws.wmflabs.org","http":{"paths":[{"backend":{"serviceName":"hub","servicePort":8081}}]}}]}} creationTimestamp: "2020-06-19T11:24:50Z" generation: 1 name: hub namespace: prod resourceVersion: "5008699" selfLink: /apis/extensions/v1beta1/namespaces/prod/ingresses/hub uid: 6c1fed46-acdf-457e-830f-928980501429 spec: rules: - host: paws.wmcloud.org http: paths: - backend: serviceName: hub servicePort: 8081 - host: paws.wmflabs.org http: paths: - backend: serviceName: hub servicePort: 8081 status: loadBalancer: {} kind: List metadata: resourceVersion: "" selfLink: ""
That all needs a bit of love one way or another.
@aborrero I fixed it! I had to delete the ingress object named hub, which may have been manually created. Then I had to re-enable the ingress in jupyterhub.
It's name is jupyterhub. We have a working prototype!!!!
I'm actually going to re-open this because helm still doesn't have a user impersonation option. That means that I should make maintain-kubeusers set admins to "edit" instead of "view" by default in PAWS, leaving the impersonation RBAC in place.
We can agree to just leave manual deploys to root as well. That's ok. kubectl can impersonate cluster-admin just fine for admins. Actually, I think that is ok as long as we stick to the upgrade schedules of Toolforge
That and apparently I just got one of the issues for adding --as to helm reopened https://github.com/helm/helm/issues/5303