In general I'm a fan of dynamic neighbors so happy to use it here on the Ganeti side. I don't think the security concerns are significant for this use-case.

An internal RPKI infrastructure would remove the "BGP VM advertise a different IP than it's supposed to" risk, but it's a significant task.

Agreed this would be overkill. It also won't constraint what source IPs can announce a given range, only the ASNs.

1/ AS path length

Yeah this one is tricky. We are running into the issue of BGP (and most routing protocols) being designed to find the shortest path to a given destination, and Anycast being kind of a hack.

IBGP would certainly remove one variable, but I fear we could be chasing it forever. My own feeling (sort of similar to T360772) is that we should do what's needed to ensure internet traffic is load-balanced equally amongst VMs, and not worry too much about the internal flows (let them go to the nearest instance).

2/ Migration BGP failover

BFD is the obvious one here. We could also perhaps reduce the timers down, 5/15 might be do-able (below that bfd is better). But maybe check with application owners, if most use-case is anycast perhaps a blip during moves is acceptable.

The ideal path would require upstream changes in Bird (especially for v6) - http://trubka.network.cz/pipermail/bird-users/2024-April/017580.html

I think we only have a few VM types that do BGP do we? Shouldn't be too inconvenient to have two variants of the config (depending what ganeti cluster the VMs are on).

I was playing with this a bit and hit some issues with using IPv6 link-local. The link-local is a normal /64 network, so no complications due to a /128 there. However I think there is a snag, in that the dynamic neighbors on the hypervisor side won't work without an "interface" specified. i.e. just using a statement link this:

neighbor range fe80::/64 external;

Isn't enough to make the connection work, the hypervisor logs this error, as the interface is included in the 'address' it is parsing:

ganeti2033 bird[3917127]: BGP: Unexpected connect from unknown address fe80::a800:ff:fe6b:aa1c%tap0 (port 58323)

Adding interface "tap0"; to the protocol block corrects the issue, allowing dynamic range fe80::/64 to be used, but that doesn't seem very useful - we'd need to have a separate configuration block for each VM/tap interface.

It does work if configured from the VMs global unicast to the hypervisors link-local address, however. We need to make sure the multihop statement is added on the hypervisor side (as its peer isn't on a local subnet), and not on the VM (as its peer is). Fwiw the configs I was playing with are here and here.

In terms of live migration and session re-establishment we should bear in mind that BIRD bgp sessions will use default BGP timers of 240/80. So the session will send a keepalive every 80 seconds and take up to 4 minutes to go down.

root@testvm2006:~# birdc show protocols all ganeti_v4 | egrep "timer|^[a-z|A-Z]"
BIRD 2.0.12 ready.
Name       Proto      Table      State  Since         Info
ganeti_v4  BGP        ---        up     08:55:59.568  Established   
    Hold timer:       194.377/240
    Keepalive timer:  34.664/80

If we don't use BFD we should tweak these in the config down to something more sensible.

@ayounsi just to document our chat on irc about the direct/multihop stuff.

To me the problem is that there is no way to have a neighbor in Bird which is considered "direct" or "onlink" if the neighbor IP is not on a local subnet. There is a 'direct' statement that can be added, but with such a neighbor this has no effect (BIRD still does not attempt to establish the session). Looking at the code I believe the complication may be that a "direct" neighbor is always bound to an interface, and when a neighbor cannot be associated with one based on the set of interface IPs it is much harder to work out that relationship. It is probably possible, but it would take more checks (routing table / ARP table) than are currently done, and I suspect these could vary based on underlying platform/OS. Routes and ARP entries are also liable to change much more than interface IP config, so constantly monitoring those and updating the interface/neighbor status may be difficult.

I think what might be easier to implement in BIRD would be to mark a neighbor direct if:

• The interface it is reachable on is specified in the protocol block
• The direct statement is included in the protocol block

I'm not sure if this should be the default or not tbh. Perhaps for EBGP 'direct' should be assumed unless 'multihop' is specified, so that statement may not be required. But I do wonder if there are any existing setups such a default would break. Either way finding the interface to bind to seems complicated, so providing it in the config might be an easier change in Bird. Luckily with our ganeti VMs the interface name is predictable and constant, so it ought not be a headache for us.

For now, if we are required to use multi-hop, the peer won't fail if the interface status changes. I'm not sure if there is an up/down link transition when a VM is moved, but we can use BFD to ensure quick teardown if required. So hopefully this is only a minor concern operationally.