Investigate current ingress architecture for PAWS and move to a more manageable and understandable one.
PAWS has three main ingress endpoints the jupyterhub proxy (https://paws.wmflabs.org ), the deploy-hook (https://paws-deploy-hook.wmflabs.org ) and the paws-public interface (https://paws-public.wmflabs.org ).
Before I broke the ingress controller for the PAWS cluster on May 19, each was using a different configuration. Let's use this task to attempt to bring them all into sane, documented methods of ingress.
PAWS still uses a proxy in the PAWS VPS project for public ingress but used a mix of kubelego and nginx ingress controller to provide ingress into the kubernetes pods.
In an attempt to upgrade to cert-manager, instead of kubelego, to provide certificate renewal (an expired certificate impeded the auto-deploy feature from working), I apparently broke ingress for PAWS.
After trying to fix it, I gave up and used the same workaround implemented in PAWS-beta for ingress (changing the kubernetes service type to NodePort and pointing the proxy to that on any cluster node).
That is not a great design, however, as it will fail on a single node failure.
We should investigate and deploy Kubernetes ingress for PAWS-beta and then push this to PAWS as well. Ideally, we incorporate this into the helm chart and keep it all automated to deploy.