@GTirloni this might be an interesting project for you to work with @Chicocvenancio on. It's been sitting in my "things to help do" list for far too long.

@bd808, looks interesting! I'll get up to speed and sync with Chico.

I tested an architecture with TCP load balancing (nginx stream module) in front of the k8s workers and then just used plain Ingress resources.

This seemed simpler and more in line with k8s best practices rather than rolling out our own via web proxies in Horizon.

This should make the PAWS cluster really standalone, not depending on magical Horizon webproxies (which can't handle multiple IPs and don't do health checking).

Having all traffic through a single ingress controller running on the master feels like a k8s anti-pattern to me. The master shouldn't be schedulable at all (besides the underlying API/scheduler/controller-manager processes).

Having a few LBs in front of the k8s workers is a common architecture for deploying k8s on bare metal. Once/if we have proper automation for this, it can be deployed anywhere.

If PAWS traffic grows any larger than the master can handle we'll have to re-architecture everything anyway but maybe we can cross that bridge when we get there.

There's a time/energy constraint on my part so I don't want to be a blocker here but I wanted to share my proposal. I'm happy to work with @Chicocvenancio on the steps that require admin access to the tools project to carry out his proposal with webproxies.

Sample nginx config for load balancers (missing proper health checks, etc).

/etc/nginx/conf.d/paws.conf:

stream {
  upstream paws_workers_http {
    server paws-k8s-worker-01.paws.eqiad.wmflabs:32080 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-02.paws.eqiad.wmflabs:32080 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-03.paws.eqiad.wmflabs:32080 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-04.paws.eqiad.wmflabs:32080 max_fails=3 fail_timeout=10s;
  }

  upstream paws_workers_https {
    server paws-k8s-worker-01.paws.eqiad.wmflabs:32443 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-02.paws.eqiad.wmflabs:32443 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-03.paws.eqiad.wmflabs:32443 max_fails=3 fail_timeout=10s;
    server paws-k8s-worker-04.paws.eqiad.wmflabs:32443 max_fails=3 fail_timeout=10s;
  }

  server {
    listen 80;
    proxy_pass paws_workers_http;
  }

  server {
    listen 443;
    proxy_pass paws_workers_https;
  }
}

I am not opposed to this approach, but do think it doesn't bring a lot of extra reliability to PAWS unless we add some other changes to make the PAWS k8s cluster HA and maybe add nodes to increase capacity.

As of now, the master is a SPOF and as I see it, it will continue to be with only this change.

If we can add at least a second master and make some capacity planning for PAWS peak usage (mainly during hackathons) I wholeheartedly support a more robust ingress.

Otherwise, I'd prefer the simpler ingress with k8s nodeports pointing to the master as there will be some work needed for this.

In order to simplify the proxy setup and free the existing A record at paws.wmflabs.org to be used in a HA load balancing scenario, I've deleted the paws.wmflabs.org and paws-public.wmflabs.org webproxies.

I've recreated those addresses as A records and pointed them to paws-proxy-02 where I configured TLS termination using Let's Encrypt (with automatic renewal).

This eliminated one proxy layer from this setup where it previous had 3: webproxy -> paws-proxy-02 -> k8s nginx ingress -> app

As this is now being pursued as a quarter goal in T211096 with the effort to reuse much of the design and testing work done for Toolforge k8s, there will be significant updates to this soon. I suspect that we will likely be able to use the haproxy (in Gio's model above it was nginx)->ingress model of Toolforge, using the existing front proxy to temporarily smooth the transition. So far, the cluster is all up and ready. We are close to sorting out the last steps of actually deploying a paws in parallel there. I'm also hoping that "paws beta" can become simply the beta namespace inside this cluster.

paws-public will need to move into this new cluster or it won't be able to use the correct NFS shares. I am doing a lot of updates to the paws stuff in the repo at my fork here: https://github.com/crookedstorm/paws pending pull request when it's ready to fully deploy.

I'll update the description above with what I think are the next steps, and I'll let @aborrero and @Chicocvenancio correct me 🙂

I think I'm not familiar enough with toolforge ingress to comment on that part.

Regarding the current PAWS ingress, in my view the closer we can get to managing it in kubernetes the better.

Implement an ingress with cert-manager in the new cluster (ingress + cert manager yaml manifests could be stored under https://github.com/crookedstorm/paws/tree/master/manifests or I can get them in the helm chart if we like that and you get them working)

The current z2jh chart should have this available, not too complicated to do it outside, but might be worth it to use the chart.

Actually get PAWS deployed in there before or after that first point (works in a minikube as I've got it set up, but that doesn't help us)
This will require sorting out a way to store the images either with different tags in quay.io or on our setup that has no automation (figuring out the former sounds better, though it may be tied to Yuvi's personal accounts).

The current images are in https://quay.io/organization/wikimedia-paws-prod/ I can give access to users there, or we can move to a different registry. The deploy hook automation is now working.

From a quick review, I take it we need 3 inbound communication channels, right?

• hub (the jupyterhub software itself). Current address: paws.wmflabs.org/hub/
• deploy-hook (for deploying stuff inside k8s, helm). Current address: paws-deploy-hook.tools.wmflabs.org
• paws-public (for something like sharing static links to notebooks?). Current address: paws-public.wmflabs.org/paws-public/

If this is true (please, help me understand otherwise), I can start thinking on the different options we may have.

Open questions:

• do we expose the privileged ports in haproxy? i.e, tcp/80 & tcp/443. Basically, I'm aiming to skip the front-proxy model we have in toolforge and let nginx-ingress inside the k8s do TLS.
• where to put/point the public floating IP. We need a floating IP so users in the internet can access PAWS. From older comments, I think we would like to avoid using horizon proxies, which is fine. We may need to point the floating IP to haproxy, which should work just fine.
• more to follow soon as I investigate options and draw diagrams.

I'm working on this preliminary architecture:

• haproxy is the main entry point for ingress and for the API server (like we do in tools) with one important difference. We assign a public floating IP directly to haproxy for it to handle http/https directly. No need for a dedicated front-proxy.
• haproxy for the api server is unchanged to what we do in tools. This is already implemented in the PAWS k8s servers deployment.
• haproxy "hides" the actual internal TCP port used in k8s by nginx-ingress
• since this architecture avoids an external front proxy like tools, nginx-ingress is reponsible for both TLS termination and connection upgrade HTTP -> HTTPS. @Bstorm suggested the cert-manager mechanism which I ignore right now.
• the architecture includes dedicated nginx-ingress node (a couple of them) which is something we would like to add to tools as well, see T250172: Toolforge: k8s: ingress: consider creating ingress-specific nodes for context
• the nginx-ingress configuration objects are stored in the paws repo, and according to my last comment T195217#6181662, it seems we need at least 3 main "channels" or FQDNs.

Perhaps all I wrote here was obvious to you all, but I really needed to draw a diagram an write a description for me to understand it.

Please @Bstorm and @Chicocvenancio, comments welcome.

Had an IRC chat with @Bstorm. We will try first with doing TLS termination in haproxy to avoid the apparent cert-manager rabbit hole. We can just reuse acme-chief for that.

Mentioned in SAL (#wikimedia-cloud) [2020-06-04T12:02:09Z] <arturo> created 2 medium VM instances: paws-k8s-ingress-1 and paws-k8s-ingress-2 with haproxy anti-affinity (T195217)

Mentioned in SAL (#wikimedia-cloud) [2020-06-04T12:04:50Z] <arturo> created paws-k8s-ingress puppet prefix and add the role::wmcs::paws::k8s::worker role (T195217)

Mentioned in SAL (#wikimedia-cloud) [2020-06-04T12:18:45Z] <arturo> bootstrapped paws-k8s-ingress nodes, added them to the k8s cluster (T195217)

Mentioned in SAL (#wikimedia-cloud) [2020-06-04T14:16:17Z] <arturo> added node taints to ingress nodes: kubectl taint nodes paws-k8s-ingress-1 ingress=true:NoSchedule (T195217)

Mentioned in SAL (#wikimedia-cloud) [2020-06-11T10:15:29Z] <arturo> added role (just a label) for ingress nodes: kubectl label node paws-k8s-ingress-1 kubernetes.io/role=ingress (T195217)

Change 604649 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: relocate nginx-ingress config from kubeadm

https://gerrit.wikimedia.org/r/604649

Mentioned in SAL (#wikimedia-cloud) [2020-06-11T11:11:06Z] <arturo> deployed nginx-ingress for some early testing (not definitive) with code https://github.com/crookedstorm/paws/commit/bee62b3fd57f9804aa27e7b8b41fde50bd93df94 (T195217)

Change 604665 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] kubeadm: rename hiera key for ingress nodes

https://gerrit.wikimedia.org/r/604665

Change 604665 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] kubeadm: rename hiera key for ingress nodes

https://gerrit.wikimedia.org/r/604665

Change 604783 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] wmcs: kubeadm: introduce different haproxy port frontend/backend

https://gerrit.wikimedia.org/r/604783

Change 604783 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] wmcs: kubeadm: introduce different haproxy port frontend/backend

https://gerrit.wikimedia.org/r/604783

Change 604789 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] wmcs: kubeadm: haproxy: introduce support for TLS termination

https://gerrit.wikimedia.org/r/604789

Change 604789 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] wmcs: kubeadm: haproxy: introduce support for TLS termination

https://gerrit.wikimedia.org/r/604789

Mentioned in SAL (#wikimedia-cloud) [2020-06-12T13:05:21Z] <arturo> live-hacking puppet tree in paws-puppetmaster-01 for T195217

Change 605217 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] paws: haproxy: fix some small issues

https://gerrit.wikimedia.org/r/605217

Change 605217 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] paws: haproxy: fix some small issues

https://gerrit.wikimedia.org/r/605217

Mentioned in SAL (#wikimedia-cloud) [2020-06-12T13:20:46Z] <arturo> created DNS zone paws.wmcloud.org (T195217)

Mentioned in SAL (#wikimedia-cloud) [2020-06-12T13:23:28Z] <arturo> DNS zone paws.wmcloud.org transferred to the PAWS project (T195217)

Mentioned in SAL (#wikimedia-cloud) [2020-06-12T13:23:53Z] <arturo> assigned the DNS zone paws.wmcloud.org (T195217)

@aborrero As we near completion, we might want to move the manifests for this into our helm chart. I can help do that when it is done as a static manifest, if we decide to. I'm hoping to get sane defaults into the helm chart in general so that someone could easily spin this up for testing and development. For that the ingress may actually be disabled by default and enabled in the helm command or something like that. Don't worry about it for now. We may put something very different in the chart for testing while keeping this outside the chart as part of the infra.

Also, sometime before PR, we might want all static yaml that are not in the helm chart in /manifests, where I left the psp yaml manifest (like just move the folder under manifests or something--I stole the word from kubeadm's static pod location). That was just an effort to contain anything we do as static yaml.

This is getting close!

Mentioned in SAL (#wikimedia-cloud) [2020-06-15T11:45:55Z] <arturo> reset wikitech user password for the service account paws-dns-manager to what is in labs/private.git/hieradata/common.yaml profile::acme_chief::cloud::designate_sync_password (T195217)

Mentioned in SAL (#wikimedia-cloud) [2020-06-15T12:18:59Z] <arturo> associate floating IP 185.15.56.57 with VM paws-k8s-haproxy-1 (T195217)

Mentioned in SAL (#wikimedia-cloud) [2020-06-15T12:20:57Z] <arturo> created DNS record paws.wmcloud.org IN A 185.15.56.57 (T195217)

Mentioned in SAL (#wikimedia-cloud) [2020-06-15T12:27:59Z] <arturo> manually created an Ingress object to test routing to the hub (T195217)

Mentioned in SAL (#wikimedia-cloud) [2020-06-15T15:59:51Z] <arturo> created DNS record deploy-hook.paws.wmcloud.org IN CNAME paws.wmcloud.org (T195217)

Change 604649 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: relocate nginx-ingress config from kubeadm

https://gerrit.wikimedia.org/r/604649

Change 605876 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] wmcs: kubeadm: decouple haproxy profile for Toolforge/PAWS

https://gerrit.wikimedia.org/r/605876

Change 605876 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] wmcs: kubeadm: decouple haproxy profile for Toolforge/PAWS

https://gerrit.wikimedia.org/r/605876

Change 605883 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] wmcs: paws: add support for HTTP->HTTPS redirection in haproxy

https://gerrit.wikimedia.org/r/605883

Change 605883 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] wmcs: paws: add support for HTTP->HTTPS redirection in haproxy

https://gerrit.wikimedia.org/r/605883

Change 605944 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] wmcs: paws: haproxy: add keepalived support

https://gerrit.wikimedia.org/r/605944

Mentioned in SAL (#wikimedia-cloud) [2020-06-16T15:23:33Z] <arturo> live-hacking paws-puppetmaster-01 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/605944 for T195217

Mentioned in SAL (#wikimedia-cloud) [2020-06-16T15:48:38Z] <arturo> change DNS record k8s.svc.paws.eqiad1.wikimedia.cloud to point to the haproxy VIP port address 172.16.1.171 (T195217)

Change 605944 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] wmcs: paws: haproxy: add keepalived support

https://gerrit.wikimedia.org/r/605944

Mentioned in SAL (#wikimedia-cloud) [2020-06-19T10:00:44Z] <arturo> enabled paws.wmflabs.org and *.paws.wmflabs.org as valid ingress domains (acme-chief TLS cert, haproxy, etc) (T195217)

I just sent a pull request to our devel repo: https://github.com/crookedstorm/paws/pull/1

@aborrero I have commented! https://github.com/crookedstorm/paws/pull/1

Btw, we got a mirror going for the upstream repo, so in the future, referencing the Phab task in a commit to an upstream branch will show up as a mention in the task :) Comments on a PR won't, of course.

Got ingress working in our test cluster by deleting the older, possibly manually created?, hub ingress. It may have been from an old test deploy. Now the ingress objects managed by jupyterhub work (and is named "jupyterhub" in the prod namespace).

If we change the hostname for the hub, we need to change it https://github.com/crookedstorm/paws/blob/master/paws/values.yaml#L170 here

Mentioned in SAL (#wikimedia-cloud) [2020-06-24T23:18:03Z] <bstorm> added A record for *.paws.wmcloud.org to public and hub to use T211096 T255997 T195217

Ok, that's a change in the direction I was originally following.

BTW, why are we using hub.paws.wmcloud.org/hub vs simply paws.wmcloud.org/hub? Why do we need the extra sub-domain here?

@aborrero We did it because of T255997#6250341

It would allow cookie isolation from the public subdomain, which is a really good idea.

In T195217#6256003, @aborrero wrote:

Ok, that's a change in the direction I was originally following.

Do you mean the hackiness of using a wildcard subdomain (because that might not be great)? Or do you mean the additional subdomain for jupyterhub? We can redo it again in any direction needed for the final, communicated version. I'm not committed to the specifics here just getting it all working except the idea of keeping the subdomain for the hub different from the public one for cookie isolation reasons. It can all be changed for the final round (especially since the cut-over requires redirection and a communication plan), but having a working PoC is still useful for knowing how far along it all is on my end as well.

Before I deleted it, amusingly, the ingress was trying to get the cert for deploy-hook:

W0625 20:47:38.848545       7 controller.go:1102] Error getting SSL certificate "prod/kubelego-deploy-hook-paws": local SSL certificate prod/kubelego-deploy-hook-paws was not found. Using default certificate

Because it still has the annotation for kubelego.

So this might finish things off here and for paws-public, though there may be a better way to do this? https://github.com/crookedstorm/paws/pull/3
I'm interested in both @bd808 and @aborrero's thoughts here.

Redirects (currently with 302 since all it really is for is paws-public and the hub login page) work with ingresses I just deployed on the test cluster.

bstormWMF2026:~ bstorm$ http -F  paws.wmcloud.org/paws/hub/login  Host:paws.wmflabs.org
HTTP/1.1 200 OK
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html
Date: Tue, 30 Jun 2020 22:33:44 GMT
Server: nginx/1.13.9
Strict-Transport-Security: max-age=15724800; includeSubDomains;
Transfer-Encoding: chunked
Vary: Accept-Encoding
access-control-allow-headers: accept, content-type, authorization
content-security-policy: frame-ancestors 'self'; report-uri /paws/hub/security/csp-report
etag: W/"1cf92188cf3d4ad93d174fba37de72a7fa86c964"
x-jupyterhub-version: 0.9.0

<!DOCTYPE HTML>
<html>

<head>
    <meta charset="utf-8">

    <title>JupyterHub</title>
    <meta http-equiv="X-UA-Compatible" content="chrome=1">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">

The above is redirecting to https://hub.paws.wmcloud.org/hub/login (stripping out the additional paws in the old URLs). So the general idea is that a pair of simple ingress objects will redirect any users who connect on the old URLs to the new once DNS for the old points to the frontend of this ingress setup.

Mentioned in SAL (#wikimedia-cloud) [2020-06-30T23:00:53Z] <bstorm> added paws-public.wmflabs.org to the alt-names for acme-chief, which broke it until we hand off the zone to the paws project <sorry!> T195217 T255997

The cert is fixed now and includes paws-public.wmflabs.org as an alt-name.
Now cutover will just mean changing the A record after final rsync.

Mentioned in SAL (#wikimedia-cloud) [2020-07-06T21:41:28Z] <bstorm> deployed ingress to redirect paws.wmcloud.org to the wikitech doc page T195217

I believe this is done but for the docs.

Change 604782 abandoned by Arturo Borrero Gonzalez:
[operations/puppet@production] wmcs: kubeadm: introduce different haproxy port frontend/backend

Reason:
No longer required. The haproxy profile for Toolforge and PAWS aren't shared in the common kubeadm class. Instead, we deduplicated the source code for both projects, given PAWS haproxy setup is a bit more complex than Toolforge one (does TLS termination, etc).

https://gerrit.wikimedia.org/r/604782