Context's context : https://phabricator.wikimedia.org/phame/post/view/312/ganeti_on_modern_network_design/
and T300152#9502030
Context :
Currently, when starting a VM in a routed Ganeti cluster, a isc-dhcp-relay daemon is brought up to relay DHCP queries between the VM and the configured install server.
This is done on that line: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/templates/ganeti/net-common.erb#205
Unfortunately isc-dhcp-relay requires to specify the -U (dhcp server facing interface) and -i (vm facing interface) for our use-case. Failing from doing so works in theory, but won't set the giaddr to the node's external interface. Instead it will use the VM facing IP.
In other circumstances, that would make sens, but to have isc-dhcpd hand over /32 IPs we need to have giaddr set to the node's IP address.
2025-06-13T10:01:54.400276+00:00 install7002 dhcpd[257748]: DHCPDISCOVER from aa:00:00:50:bc:fb via 10.140.2.1: network 10.140.2.0/24: no free leases
2025-06-12T11:44:34.692886+00:00 install7002 dhcpd[173704]: DHCPDISCOVER from aa:00:00:90:40:c0 via 10.140.0.11 2025-06-12T11:44:34.693070+00:00 install7002 dhcpd[173704]: DHCPOFFER on 10.140.2.7 to aa:00:00:90:40:c0 via 10.140.0.11
This has been working fine... ...until now.
When setting up magru, we eventually ended up in the case where the install server (running the DHCP server) was on the same hypervisor than the VM that was being created.
In that case, the reply from the install server logically enters from the install server's tap interface, and not from the expected eno12399np0 interface defined using the -U flag of dhcrelay
This causes dhcrelay to ignore the reply.
2025-06-13T07:43:06.703195+00:00 ganeti7001 dhcrelay: Forwarded BOOTREQUEST for aa:00:00:50:bc:fb to 195.200.68.100 2025-06-13T07:43:06.703265+00:00 ganeti7001 dhcrelay: Dropping reply received on tap1
In the short term, to finish setting up magru, one of the possible workaround is to migrate the install server or the VM being setup to a different hypervisor of the same cluster, or manually run the proper dhcrelaycommand.
There are multiple ways of fixing it.
For example we could run a bash or python script at VM creation time, to check if the configured install server is on the same hypervisor then get its tap interface for dhcrelay configuration. But this seems a bit brittle.
I think the most reliable way is to have a better DHCP relay daemon.
I looked around at the options once more, and a few stand out:
- isc-dhcp-relay is not maintained anymore, so no more work should go into it
- py-dhcp-relay is old and depends on even older libs, which might make it challenging to run on Debian Bookworm.
- https://github.com/Mirantis/dhcp-relay seems like a good candidate too, but not maintained anymore
- https://packages.debian.org/stable/dhcp-helper not sure if still maintained. This does support listening on all interfaces, but doesn't support setting a custom giaddr.