Page MenuHomePhabricator

Improve how virt networks are configured in cloudgw
Closed, ResolvedPublic

Description

The cloudgw configuration for virt networks[0] is a bit of a mess, as the system was initially set up with a single v4-only network for Cloud VPS VMs and an another network for floating IPs. Over the years it has grown organically to support features like multiple networks and floating IP pools, IPv6, and most recently tenant networks with no internet connectivity (T394099)

The end result of that is a config that is spread across multiple manually maintained Hiera keys all using slightly different formats:

profile::wmcs::cloudgw::virt_subnets_cidr:
  # old legacy vlan-based subnet
  - 172.16.0.0/21
  # vxlan dualstak subnet
  - 172.16.16.0/21
  # vxlan ipv4 only subnet
  - 172.16.8.0/21
profile::wmcs::cloudgw::virt_subnets_cidr_v6:
  - 2a02:ec80:a000::/55
profile::wmcs::cloudgw::virt_internal_subnets_cidr:
  # octavia-lb-mgmt-net
  - 172.16.24.0/24
profile::wmcs::cloudgw::virt_floating: [185.15.56.0/25]

I propose transforming this to a single array with all of the different networks listed, perhaps something like this:

profile::wmcs::cloudgw::virt_subnets:
  # VLAN/legacy
  - networks:
      - 172.16.0.0/21
    type: default
  # VXLAN/IPv4-only
  - networks:
      - 172.16.8.0/21
    type: default
  # VXLAN/IPv6-dualstack
  - networks:
      - 172.16.16.0/21
      - 2a02:ec80:a000:1::/64
    type: default
  # octavia-lb-mgmt-net
  - networks:
      - 172.16.24.0/24
      - 2a02:ec80:a000:100::/64
    # no internet connectivity
    type: internal
  # cloud-eqiad1-floating
  - networks:
      - 185.15.56.0/25
    type: floating

While this has some benefits on its own (namely, a format that's a bit more friendly to work with as it resembles more the format used for the same data in other places), the main benefit is that this is relatively easy to transform to a format with more than one place to route the networks to, which'll be needed for Toolforge-on-Metal. This is even more of a hypothetical example and not a definite final version, but just as an example the end goal could be something like this:

profile::wmcs::cloudgw::virt_interfaces:
  vlan1107:
    own_ip4: 185.15.56.234
    # remaining configuration to bring the interface up moved here ...
  vlanNNNN:
    # configuration for the toolforge-on-metal interface here ...
profile::wmcs::cloudgw::virt_targets:
  neutron:
    peer:
      interface: vlan1107
      address4: 185.15.56.238
      address6: 2a02:ec80:a000:fe04::2:1
    networks:
      # VLAN/legacy
      - networks:
          - 172.16.0.0/21
        type: default
      # VXLAN/IPv4-only
      - networks:
          - 172.16.8.0/21
        type: default
      # VXLAN/IPv6-dualstack
      - networks:
          - 172.16.16.0/21
          - 2a02:ec80:a000:1::/64
        type: default
      # octavia-lb-mgmt-net
      - networks:
          - 172.16.24.0/24
          - 2a02:ec80:a000:100::/64
        # no internet connectivity
        type: internal
      # cloud-eqiad1-floating
      - networks:
          - 185.15.56.0/25
        type: floating
  toolforge:
    peer:
      interface: vlanNNN
      # ...
    networks:
      # ...
      - networks: [192.0.2.0/24]
        type: default

[0]: currently: subnets used by Cloud VPS virtual networks (Neutron), soon also including Toolforge-on-Metal related networks

Event Timeline

taavi added subscribers: cmooney, fgiunchedi.

/cc @cmooney @fgiunchedi, just wanted to get your opinion on whether this looks reasonable or whether I'm missing something obvious here before I start writing the patches for this.

Looks good to me! Definitely good to refactor these bits

@taavi broadly this looks good to me, nicely done.

I wonder how binding interfaces to VRFs works here? Might it make sense to also have a key in profile::wmcs::cloudgw::virt_interfaces for each virtual interface which can define if it's a member of a VRF? (perhaps that's covered in "remaining configuration to bring the interface up moved here"?).

Certainly I'm not opposed to this at all, I think I probably need to have some bits explained as to how it all fits together but seems sane.

Change #1211667 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: cloudgw: Configure individual v6 networks

https://gerrit.wikimedia.org/r/1211667

Something I wanted to add: I'm not very familiar with that part of the puppet codebase though I was wondering if we can start referring to the networks and their attributes by name. It will enable us to have puppet code e.g. "give me the subnets for VXLAN/IPv6-dualstack, either v4 or v6 or both", in other words get to more understandable and manageable code/understanding. Ditto for other attributes/properties for a given network like its gateway, etc. Let me know what you think!

Change #1211667 merged by Majavah:

[operations/puppet@production] hieradata: cloudgw: Configure individual v6 networks

https://gerrit.wikimedia.org/r/1211667

taavi triaged this task as High priority.Dec 1 2025, 11:40 AM

Change #1213436 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs::cloudgw: Implement new virt network config structure

https://gerrit.wikimedia.org/r/1213436

Change #1213436 merged by Majavah:

[operations/puppet@production] P:wmcs::cloudgw: Implement new virt network config structure

https://gerrit.wikimedia.org/r/1213436

Something I wanted to add: I'm not very familiar with that part of the puppet codebase though I was wondering if we can start referring to the networks and their attributes by name. It will enable us to have puppet code e.g. "give me the subnets for VXLAN/IPv6-dualstack, either v4 or v6 or both", in other words get to more understandable and manageable code/understanding. Ditto for other attributes/properties for a given network like its gateway, etc. Let me know what you think!

That data is sort of already in Puppet in modules/network/data/data.yaml - if we want to make the network stuff re-usable by name then we should be extending that instead of building yet another thing.