I'm considering the possibility of setting up a separate project for PAWS, with its own k8s cluster.
The tools k8s cluster is custom and specialized for making a specific use case easy. It comes with lots of custom restrictions and what not to make the lives of the tools admins more of a hell than it already is. Such restrictions include uid restrictions, lack of root, restrictions on where docker images can be pulled from, what hostpaths could be mounted, which k8s objects could be created, etc. These customizations + requirement to not break everyone also means the k8s version lags behind a bit, and upgrading isn't super easy.
For PAWS, we could possibly switch to just setting up our own project, and setting up a k8s cluster there. The advantages of this would be:
- Easy to set up and upgrade, since there's only one user of the cluster (PAWS) and can use standard upstream k8s installation methods (kubeadm, specifically)
- We can just easily reuse https://z2jh.jupyter.org, which is now an upstream project that has advanced far beyond PAWS (although it was originally based on PAWS, we never upgraded PAWS because it required new kubernetes version features + other blockers based on the tools cluster's customization). This gets PAWS users new features quickly
- PAWS going crazy does not affect other tools users, and vice versa.
- It becomes a lot less maintenance for me, since it much more closely resembles 'just another k8s cluster'.
The disadvantages would be:
- Less efficient resource usage. I'm not entirely sure about this, since we're on OpenStack anyway.
- A new project would need to also have NFS. I am actually open to running my own NFS setup with a couple of VMs (FAMOUS LAST WORDS?!) if this is a big problem.
- Less pooled resources put into the tools k8s cluster, since I'd be working mostly on this other cluster. Compare with #4 above tho.