Management vs Control-Plane Traffic
The current WMCS physical network setup (i.e. outside the OpenStack / Neutron) is documented here.
Broadly speaking cloud hosts have two connected NICs, one belonging to the "production" WMF realm, and one belonging to the "cloud" realm. The recent changes have seen this isolation increased by introducing VRFs to the cloud switches to support routing and failover at layer-3 while retaining the required isolation.
Ideally the "production realm" link from cloud hosts would only be used for management functions, including:
- Communicating with install* hosts during reimage (DHCP, pulling OS image, apt packages etc.)
- Scraping from WMF prometheus or other monitoring
- Running cookbooks from cumin hosts
- SSH access to the host OS
Right now, however, the production realm / private network is also used by various cloud services / daemons for communication. I am not familiar with all of these, but for instance RabbitMQ traffic goes through the production realm, as does Ceph "public network" traffic from clients to the ceph cluster nodes.
New Network
Logically it makes more sense, I think, for traffic between cloud hosts, for cloud services, to remain in the cloud realm / vrf. So basically I'd propose that a new network/subnets be created, with private IPv4 addressing and unrouted IPv6 global unicast (if v6 required), to serve as transport for such protocols. This network would be placed into the existing cloud vrf on those switches, or a separate VRF if a strong requirement for isolation from cloud-instance or cloud-storage networks is identified.
Such a setup might also reduce the requirement for certain cloud services to sit on public vlans. It appears that certain services, like those running on the cloudcontrol and cloudrabbit hosts, are currently built to use public IPs to ensure cloud instances (VMs) can communicate with them. Instances are blocked from communicating directly with private WMF space otherwise.
If that is in fact the case it would seem to be a waste of public IPv4 space, and potentially opens up an attack surface (services running on public IPs and thus routable from the internet) when it is not required. The cloudgw nodes could instead be connected to the proposed new network, and route traffic coming from the cloud realm directly to services running on it, without leaving the cloud realm / vrf. Or cloudnet hosts could do this depending on where WMCS team felt it would best sit.
802.1q handoff
Connecting a third physical NIC to each cloud host is not practical for a cost and operational point of view. So probably to connect to this third "control" network we would want to handoff from datacenter switches to hosts as "vlan trunks", logically segmenting the different vlans through the use of 802.1q vlan tags.
That may require updates to what facts we expose in Puppet (see T296832), to allow for automatically setting switch ports to trunk mode with the correct vlan list. We also need a mechanism to define tagged interfaces on the cloud hosts. Our LVS hosts already configure sub-interfaces via puppet, see here and here, which might be an example to follow.
Conclusion
I'm opening this task to solicit feedback on the idea. Comments are more than welcome. I appreciate migrating from the current setup might be a long or tricky job, but if we can at least explore whether the end result would be desirable we can then consider if it is practical to implement or get there.
Overall I think it definitely makes sense to have this kind of traffic separated from the management stuff. In the long run I think it'd make things easier and more flexible for all teams.