/cc @Andrew @cmooney

Thanks for the task @taavi !

Option 1 is nice and neat, but as you say there is a lot of work to do in terms of building a new provisioning/mgmt stack within cloud-private.

Option 2 is where I imagined this would go after having a quick chat with Alexandros about it a week or so back.

I assume (but have not yet checked that) the K8s CNI used (Calico?) could be directed to use a secondary interface for all cluster traffic, leaving the primary interface for management purposes only.

Yes this is a question I am also unsure about. An ip rule could be used similar to on the cloudservice nodes to select one of two routing tables, each with a default route towards the appropriate interface. But I know k8s networking is complicated and uses many kernel features, so I'm not sure if we can add stuff like that ourselves without conflicting with Calico.

.. if there's even a reasonable way to port the current bastion hosts to this model. I can't immediately think of an approach to isolate those that gives me the confidence that I would want from such a system.

I'm not that familiar with the existing cloud bastion hosts. Happy to discuss though if you want to ping me on irc.

External Access

The other thing we need to consider is external access from things running in the new K8s PODs. Do they need to talk to things outside the cloud realm? The model used for the OpenStack VMs - whereby the traffic is either tunneled or directly sent at L2 to the cloudnet hosts, then handed off on a private vlan to the cloudgw's - is probably not an option? Off the top of my head two approaches we could take then are:

1. Direct public IP routing to the new K8s hosts
   • If Calico originates public IP addresses (v4 or v6) they can be routed directly in the cloud-private VRF and connect out to the internet
2. Use the cloudgw or another pair of firewalls/NAT boxes to support external connectivity for the new hosts
   • This kind of means we'd need another vrf on the switches (similar to we considered before), where the default route goes to these NAT boxes rather than directly to the CRs.

In T407140#11269245, @cmooney wrote:

I assume (but have not yet checked that) the K8s CNI used (Calico?) could be directed to use a secondary interface for all cluster traffic, leaving the primary interface for management purposes only.

Yes this is a question I am also unsure about. An ip rule could be used similar to on the cloudservice nodes to select one of two routing tables, each with a default route towards the appropriate interface. But I know k8s networking is complicated and uses many kernel features, so I'm not sure if we can add stuff like that ourselves without conflicting with Calico.

One thing to mention here - we only really need to care about the default route. We can announce all the specific cloud-private networks (any service IPs) to the K8s hosts over BGP, so they will learn those dynamically via the second interface. The only bit we need to work out is the default route.

I'm pretty sure that the next step (or /a/ next step) is to convert the servers in T406630 into a ganeti cluster so that we can create VMS with their feet in both realms. @cmooney, does that seem possible, or is our existing ganeti code incompatible with a dual-network approach?

In T407140#11272555, @Andrew wrote:

I'm pretty sure that the next step (or /a/ next step) is to convert the servers in T406630 into a ganeti cluster so that we can create VMS with their feet in both realms. @cmooney, does that seem possible, or is our existing ganeti code incompatible with a dual-network approach?

Yeah the existing ganeti code expects only a single connected interface. We also need to think about what the networking within the VMs would look like.

I expect we should follow the routed ganeti model. In which case we can probably have the same/similar network setup on one of these ganeti hosts as we plan for the K8s nodes. And in the same way a K8s POD does not know about multiple realms/interfaces, a VM can have a single interface (with the ganeti host acting as gateway, and controlling what traffic is allowed and what traffic gets sent where).

Definitely gets a little tricky though. Let's start with defining what the VMs we expect to run are and what they need to be able to talk to and take it from there.

I'm no longer sure that we want a ganeti cluster vs. just k8s control nodes. I think clarity will emerge about that as we fill in https://phabricator.wikimedia.org/T407299

In T407140#11281717, @Andrew wrote:

I'm no longer sure that we want a ganeti cluster vs. just k8s control nodes. I think clarity will emerge about that as we fill in https://phabricator.wikimedia.org/T407299

Ok thanks Andrew. Some of what we have to consider will be the same whether the control nodes are bare metal or on a ganeti VM. But it's one less layer of things to consider if we're not doing the VMs.

If we want to attempt the everything-on-containers plan that Alex is promoting, the next step is probably to make a wee k8s cluster out of those three nodes. I'm not clear on what's needed to set up the BGP bits here; should I just apply appropriate puppet roles and then hand things off to @cmooney ?

The initial networking questions in the task description still need deciding. It seems like we are leaning towards option 2, although I think this should have a new set of networks instead of using cloud-private for user traffic. Plus we'll need to allocate IP space for cluster pods and services.

Just to note down what we discussed on our meeting about this:

1. We will need option 2
   • To have connectivity to WMF prod for ssh, puppet, monitoring, reimage etc
   • To have connectivity to a separate network (unreachable from WMF prod) for POD traffic
2. The PODs should not have direct access to things within the cloud-private networks, such as OpenStack APIs
   • A new set of vlans, in a new VRF/routing-instance on the cloud switches can provide this isolation (T409309)
   • But potentially Calico can be configured with sufficient rules/filters to control what PODs can access, and hosts could be in cloud-private
3. The cloudgw should act as a gateway/firewall between networks
   • As it currently does between the cloud-private and neutron networks
   • If we have a new VRF it will need a leg in that and can forward permitted traffic between it and the others
   • We should consider any large traffic flows, potentially NFS mounts, and try not to have them traverse the single active cloudgw
4. The two-nic setup should be manageable similar to how we do it on the cloudservice nodes
   • The tricky part here is managing two default routes - one via the 10.x wmf prod network, one via the private one
   • Our complication may be that adding kernel rules to these hosts to control this would conflict with how Calico is managing the host networking
   • Potentially Calico itself has some configuration options we can use to create the rules we need without touching the kernel directly

cloudservice setup

On the last point this is what we do on the cloudserivce nodes to support a default on both interfaces. Regular default route is on the 10.x network to wmf prod realm:

cmooney@cloudservices1005:~$ ip -4 route show default 
default via 10.64.150.1 dev ens3f0np0 onlink

But we have some ip rules in place which will select another network if the packets are sourced from the service loopbacks the host has on lo (these could be viewed similar to POD IP ranges on the k8s hosts)

cmooney@cloudservices1005:~$ ip rule show 
0:	from all lookup local
32764:	from 185.15.56.162 lookup cloud-private
32765:	from 172.20.255.1 lookup cloud-private
32766:	from all lookup main
32767:	from all lookup default

cmooney@cloudservices1005:~$ ip route show table cloud-private 
default via 172.20.2.1 dev vlan1152

As I said above the important thing is we don't conflict with Calico. The best solution is potentially if Calico allows us to configure something similar.

Internet access

The one thing we didn't discuss that I'm interested to know is what external internet connectivity requirements these new hosts or PODs may have. Currently in cloud-private the default route goes directly out to our core routers, and no NAT is being done (the cloudservice nodes source public traffic directly from a public loopback for instance).

I guess the question is whether that makes sense here, or if instead we need to have a default going to the cloudgw (so it can do NAT or similar).

Thank you @cmooney for the summary, I'll add a few thoughts I had while working on the Toolforge on Metal project design document.

During the phase 0 / POC the bare metal hosts in theory will not require access to cloudvps VMs (i.e. cloud-private), however if the POC is successful we'll be starting the migration which will require access to tools project VMs, roughly speaking:

• phase 1
  • all tools eventually will run on ToM k8s cluster, while accessing existing stateful/read-write services in tools (e.g. NFS, toolsdb, elastic, redis)
• phase 2
  • we will be migrating the above services one by one to their ToM counterparts, including data migration and flipping tools to use said services

All of that to say that some services might be high traffic such as NFS, also worth noting that while still TBD the traffic might be originating from the hosts themselves and not necessarily the k8s pods. Happy to brainstorm more and see if the above changes any consideration

Ok thanks @fgiunchedi for the info.

I think that seems doable. As per the sub-task about a VRF I think that will be needed. And route leaking is not something I really want to do, so we will probably need the "cloudgw" to route traffic between the new VRF and the exsiting cloud-private / Openstack networks.

In T407140#11348677, @fgiunchedi wrote:

• phase 1
  • all tools eventually will run on ToM k8s cluster, while accessing existing stateful/read-write services in tools (e.g. NFS, toolsdb, elastic, redis)

What is on the NFS shares? Are these NFS shares in use at the moment by the current VM-based cluster? And if so what is the typical peak throughput/pps those shares have to support?

Reason I ask is that if that traffic needs to flow through the cloudgw (or other centralised "firewall" that mediates traffic between the two zones) then it could be a problem if it was very high bandwidth.

In T407140#11383672, @cmooney wrote:

Ok thanks @fgiunchedi for the info.

I think that seems doable. As per the sub-task about a VRF I think that will be needed. And route leaking is not something I really want to do, so we will probably need the "cloudgw" to route traffic between the new VRF and the exsiting cloud-private / Openstack networks.

Makes sense, thank you for clarifying

In T407140#11348677, @fgiunchedi wrote:

• phase 1
  • all tools eventually will run on ToM k8s cluster, while accessing existing stateful/read-write services in tools (e.g. NFS, toolsdb, elastic, redis)

What is on the NFS shares? Are these NFS shares in use at the moment by the current VM-based cluster? And if so what is the typical peak throughput/pps those shares have to support?

Reason I ask is that if that traffic needs to flow through the cloudgw (or other centralised "firewall" that mediates traffic between the two zones) then it could be a problem if it was very high bandwidth.

The NFS shares host tools data and scratch space, specifically:

scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud:/srv/scratch  3.0T  1.3T  1.6T  44% /mnt/nfs/secondary-scratch
tools-nfs.svc.tools.eqiad1.wikimedia.cloud:/srv/tools/home       10T  7.2T  2.3T  76% /mnt/nfs/labstore-secondary-tools-home
tools-nfs.svc.tools.eqiad1.wikimedia.cloud:/srv/tools/project    10T  7.2T  2.3T  76% /mnt/nfs/labstore-secondary-tools-project

There are also NFS read only shares towards clouddump hosts, though those are on the public IP space and I believe those won't go through cloudgw.

re: network usage, good question; the easiest I think is checking the instances stats themselves:

• cloudinfra-nfs, minimal network usage: https://grafana.wmcloud.org/goto/HfOPZamvR?orgId=1
• tools-nfs, higher usage though nothing unmanageable I think: https://grafana.wmcloud.org/goto/7j78W-ivg?orgId=1

HTH!

In T407140#11391233, @fgiunchedi wrote:

There are also NFS read only shares towards clouddump hosts, though those are on the public IP space and I believe those won't go through cloudgw.

All traffic in and out of the Neutron virtual networks does go through cloudgws at the moment, and that includes traffic to the dumps shares.

In T407140#11391233, @fgiunchedi wrote:

The NFS shares host tools data and scratch space, specifically:

scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud:/srv/scratch  3.0T  1.3T  1.6T  44% /mnt/nfs/secondary-scratch
tools-nfs.svc.tools.eqiad1.wikimedia.cloud:/srv/tools/home       10T  7.2T  2.3T  76% /mnt/nfs/labstore-secondary-tools-home
tools-nfs.svc.tools.eqiad1.wikimedia.cloud:/srv/tools/project    10T  7.2T  2.3T  76% /mnt/nfs/labstore-secondary-tools-project

There are also NFS read only shares towards clouddump hosts, though those are on the public IP space and I believe those won't go through cloudgw.

re: network usage, good question; the easiest I think is checking the instances stats themselves:

• cloudinfra-nfs, minimal network usage: https://grafana.wmcloud.org/goto/HfOPZamvR?orgId=1
• tools-nfs, higher usage though nothing unmanageable I think: https://grafana.wmcloud.org/goto/7j78W-ivg?orgId=1

This is great info @fgiunchedi thanks! Yeah that usage is minimal it won't give us any problems at all. In my mind I expected multiple tens of Gbps that doesn't warrant thinking about.

Not sure if it has been discussed but what do you think of using Calico's VXLAN or IP-IP overlay ?
For what I understand it seems like the perfect solution for this usecase, and is quite similar to what we would have wanted of CloudVPS if we were to redo it from scratch.

Nodes stay in the per rack cloud-hosts vlan (which are located in the cloud racks) so they have access to prod systems for provisioning/monitoring (but with extra firewalling in case of pod escape), like the current cloudvirt hosts.
But instead of doing network separation using a complex VRF and ip rules (which are often a long term pain to maintain), we let the PODs talk to each other though Calico's overlay.
To be investigated more in details but we should also be able to make any cloudgw part of the overlay for NAT or external access (like an external LB), as well as any production resource needed directly by the PODs that can't be exposed through the internet (eg. NFS). For that last one, it should of course happen on a case by case basis with a thorough security evaluation.

This diverges from the production BGP setup of k8s as in production the PODs traffic is trusted.

In T407140#11391938, @ayounsi wrote:

But instead of doing network separation using a complex VRF and ip rules (which are often a long term pain to maintain), we let the PODs talk to each other though Calico's overlay.

Conceptually its the same amount of complexity I think, but the VRF/ip rules bit etc is pushed to the host and perhaps Calico just "does it all" for us so we don't need to care. So simpler to deploy/manage.

To be investigated more in details but we should also be able to make any cloudgw part of the overlay for NAT or external access (like an external LB), as well as any production resource needed

Yeah that bit is the trick, how we provide the external gateway via the Calico virtual network. Can the CloudGW be part of that cluster acting as a VTEP for traffic sent to it by Calico on the new TOM hosts? Or do we need another node similar to the CloudNet / neutron hosts which is running Calico itself and does the decap, handing off to CloudGW on a dedicated vlan like cloudnet?

Following up from Lisbon: there are essentially two options available wrt network implementation, @cmooney has kindly offered to outline both each with their own pros and cons to make it easier to come to a conclusion

@fgiunchedi I got a few minutes to do a bit of a brain dump here - please see this Google Doc.

Overall my preference is to make this as much of a "clean slate" as possible. So my preference if for Option 1, making a new VRF for the Toolforge hosts, and using the existing cloudgw's to provide access from it to all other destinations (Internet via NAT, WMF prod for mgmt, existing OpenStack etc).

Option 2 is without doubt an option. It still relies on the cloudgw's for connectivity, but here the k8s hosts tunnel traffic to the cloudgw over the existing cloud-hosts network. It can be deployed immediately as the wire-line networks are already there, however the WMCS SREs will need to determine the exact k8s config to make it work on the hosts (we of course can advise).

Should this ticket still be open? is there any work here still relevant irrespective of ToM project?