https://github.com/yuvipanda/paws/pull/27 this should take care of all necessary configuration for Kubernetes itself, there is still the matter of sending traffic to the kubernetes cluster (at any node, the master makes most sense to me).

Currently, this is using the paws-proxy-01 instance, which then has a nginx to route the appropriate backend. I think we should ditch this and use the web-proxy available to horizon to setup the ingress, as is currently done in PAWS-beta. This also requires a security group change, it can be a copy of security group 932 (@toolsbeta, IE, ports 32611-32612 accessible from 10.0.0.0/8).

There is still the matter of paws-public before being able to completely ditch the paws-proxy-01 instance (and the whole VPS project). I'll look into this tomorrow.

The move from DNS pointing to paws-proxy-01 to web-proxies through horizon might involve downtime if something goes wrong, (and needs to be done in the tools VPS project). How should we handle this?

@Bstorm's discovered paws-public is basically a kubernetes tool running in toolforge.

I propose the following tasks:

• Edit the security groups in the tools vps project to allow access to the ports 32611 and 32612 from the tools-proxy-01 (can be 10.0.0.0/8)
• Remove the web-proxy for paws.wmflabs.org in the paws vps project
• Create a web-proxy in the tools vps project pointing to tools-paws-master-01 at port 32611 for paws.wmflabs.org
• Remove the web-proxy for paws-deploy-hook.wmflabs.org in the paws vps project
• Create a web-proxy in the tools vps project pointing to tools-paws-master-01 at port 32612 for paws-deploy-hook.wmflabs.org
• Edit the paws redirection in toolforge to also redirect to paws-public if the host is paws-public.wmflabs.org (could this live in the redirects vps project?)
• Create a web-proxy in the tools vps project pointing to tools-proxy-01 for paws-public.wmflabs.org (will this work? the idea is to have requests for paws-public.wmflabs.org be dealt with by the general toolforge rules)

With that the paws vps project can be deleted fully and we will have an easier to reason about ingress for each component. IE, web-proxies pointing to nodeports of the kubernetes cluster and a redirect to a toolforge tool.

As noted before, this may cause (hopefully brief) downtime for paws.wmflabs.org.
The tools vps project tasks will need (toolforge)root time to be executed.

Adding the team project here to get on their radar.

Noting jupyterhub 0.9 has been released. This task is now the main blocker for deploying changes to PAWS.

There is still some work to be done, but I should be able to go through it next week.

Main tasks left:

• T120352
• organizing the mess of diff from PAWS-beta to PAWS
• T197248
• T197514

Mentioned in SAL (#wikimedia-cloud) [2018-06-20T22:03:53Z] <chicocvenancio> updated nginx-ingress-controler to sidestep T195217

@GTirloni this might be an interesting project for you to work with @Chicocvenancio on. It's been sitting in my "things to help do" list for far too long.

@bd808, looks interesting! I'll get up to speed and sync with Chico.

I tested an architecture with TCP load balancing (nginx stream module) in front of the k8s workers and then just used plain Ingress resources.

This seemed simpler and more in line with k8s best practices rather than rolling out our own via web proxies in Horizon.

This should make the PAWS cluster really standalone, not depending on magical Horizon webproxies (which can't handle multiple IPs and don't do health checking).

Having all traffic through a single ingress controller running on the master feels like a k8s anti-pattern to me. The master shouldn't be schedulable at all (besides the underlying API/scheduler/controller-manager processes).

Having a few LBs in front of the k8s workers is a common architecture for deploying k8s on bare metal. Once/if we have proper automation for this, it can be deployed anywhere.

If PAWS traffic grows any larger than the master can handle we'll have to re-architecture everything anyway but maybe we can cross that bridge when we get there.

There's a time/energy constraint on my part so I don't want to be a blocker here but I wanted to share my proposal. I'm happy to work with @Chicocvenancio on the steps that require admin access to the tools project to carry out his proposal with webproxies.

Sample nginx config for load balancers (missing proper health checks, etc).

/etc/nginx/conf.d/paws.conf:

stream {
  upstream paws_workers_http {
    server paws-k8s-worker-01.paws.eqiad.wmflabs:32080 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-02.paws.eqiad.wmflabs:32080 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-03.paws.eqiad.wmflabs:32080 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-04.paws.eqiad.wmflabs:32080 max_fails=3 fail_timeout=10s;
  }

  upstream paws_workers_https {
    server paws-k8s-worker-01.paws.eqiad.wmflabs:32443 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-02.paws.eqiad.wmflabs:32443 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-03.paws.eqiad.wmflabs:32443 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-04.paws.eqiad.wmflabs:32443 max_fails=3 fail_timeout=10s;
  }

  server {
    listen 80;
    proxy_pass paws_workers_http;
  }

  server {
    listen 443;
    proxy_pass paws_workers_https;
  }
}

I am not opposed to this approach, but do think it doesn't bring a lot of extra reliability to PAWS unless we add some other changes to make the PAWS k8s cluster HA and maybe add nodes to increase capacity.

As of now, the master is a SPOF and as I see it, it will continue to be with only this change.

If we can add at least a second master and make some capacity planning for PAWS peak usage (mainly during hackathons) I wholeheartedly support a more robust ingress.

Otherwise, I'd prefer the simpler ingress with k8s nodeports pointing to the master as there will be some work needed for this.

In order to simplify the proxy setup and free the existing A record at paws.wmflabs.org to be used in a HA load balancing scenario, I've deleted the paws.wmflabs.org and paws-public.wmflabs.org webproxies.

I've recreated those addresses as A records and pointed them to paws-proxy-02 where I configured TLS termination using Let's Encrypt (with automatic renewal).

This eliminated one proxy layer from this setup where it previous had 3: webproxy -> paws-proxy-02 -> k8s nginx ingress -> app