An idea that @aborrero and I talked about a bit at the Istanbul Hackathon would be to allocate IPv6 in Toolforge's Kubernetes cluster by namespace/tool aware blocks with the Unix group id of the owning tool encoded into the address. Imagine each namespace getting a /64 or /96 allocation where the least significant bits of the network encoded the gidNumber of the tool's account. For example tools.gitlab-account-approval's gidNumber of 55655 might turn into a network of 2001:db8::d967/96. 55655 decimal == d967 hex.

The main advantage of this system would be the ability to trace IPv6 address usage back to a particular tool without needing to maintain a lookup table for the network allocations or DHCP leases that might vary over time.

There are several things to consider here. First of all, we just cannot take a chunk of the Cloud VPS VXLAN/IPv6-dualstack CIDR and give it to Toolforge.

Instead, we would need to either:

• create another admin-owned flat network for dedicated use only by Tools/Toolsbeta
  • via tofu-infra <-- perhaps more simple to implement, but also less robust?
• create a Tools/Toolsbeta specific dualstack tenant network
  • via toolforge/tofu-provisioning <-- perhaps more complex to implement, but also more robust?

In both cases we would achieve the same:

• an IPv6 CIDR allocation that will get routed in the edge by neutron, but neutron will not 'handle' itself (i.e, no DHCP, no VM ports with addresses from this CIDR).
• this allow to delegate this IPv6 CIDR to Kubernetes inside tools/toolsbeta
• VMs for K8s worker nodes will need to explicitly allow in security groups and/or port security for this IPv6 CIDR to circulate, since usage will be outside of the accountability of Neutron.

Inside Kubernetes, I think the implementation would look like this. Mostly following the ideas from https://docs.tigera.io/calico/latest/networking/ipam/legacy-firewalls

• enable the calico IPAM plugin, if not already enabled
• define one calico IPpool resource per tool namespace (https://docs.tigera.io/calico/latest/reference/resources/ippool#sample-yaml)
  • we will encode the tool uid here in the CIDR parameter, as @bd808 described in https://phabricator.wikimedia.org/T380060#10823558
  • this can be created using maintain-kubeusers, in a similar fashion to other per-tool resources
  • note this resource itself is not namespaced, it is cluster global. We need to name them with the name of the tool account: tool-mytool1, tool-mytool2, etc.
• add an annotation to each tool namespace, for example cni.projectcalico.org/ipv6pools: '["tool-mytool1"]' in the tool-mytool1 namespace
• note that one could override this annotation on the pod level, which potentially allows one tool to run its workloads on other tool CIDR (bad) To prevent this we could take a few measures:
  • use Kyverno, or whatever policy engine, to disallow this annotation at pod level
  • introduce Network policy namespaced resources (see https://docs.tigera.io/calico/latest/reference/resources/networkpolicy) to enforce IPv6 network traffic on each tool namespace has the right CIDR with the tool uid encoded
    • and make sure the cluster policy doesn't allow users to modify this Network policy resource

Fortunately it seems we have most of the infrastructure ready to introduce all this config, (i.e no need to write custom admission controllers or anything).