Recently Moritz hit an issue when re-imaging a Ganeti host.
Background
The re-image cookbook needs to know the switch, switch port, and vlan id that a given host will send DHCP DISCOVER messages on when it boots. The switch inserts this info into the DHCP packets when it relays them to install hosts. The re-image cookbook adds a DHCP config snippet to match this information, and return the correct IP assignment in the DHCP OFFER.
Mechanics
The re-image cookbook gets this information from Netbox. Specifically it takes the primary IP of the given device, looks at what interface it is configured on, finds the switch port it's connected to, and checks the untagged vlan defined for that port.
That works fine for standard hosts with their primary IP configured directly on a physical interface. For the Ganeti hosts, however, there is a problem. During the first puppet run the ganeti_init.sh script is run, which configures two bridge devices on the host, and moves the primary IP from the physical interface to the new "private" bridge device. The server's physical interface is then made a member of the bridge, so this works fine on the data plane.
The re-image cookbook calls the puppetdb import script as part of it's execution, which takes the interface data from puppetdb and updates the device interfaces in Netbox based on that. In the case of Ganeti this causes 3 new virtual interfaces to be created, one a vlan sub-interface (eno1.xxxx), and two bridge devices (private, public). The primary IP of the server is moved in Netbox from the physical interface (linked to switch port) to the virtual "private" device.
As a result of all that, when the re-image cookbook gets the device associated with the device's primary IP, it is a "virtual" device (the private bridge device), which has no associated switch port. So getting the switch port fails.
Data Model
Our current Netbox version does not allow us to define the relationship between physical ports and 802.1q sub-interfaces. Nor does it allow the creation of "bridge" devices/interfaces, which can be the "parent" of other interfaces. We have both of these types on our Ganeti hosts.
The good news is that the ability to define both of these has been added to more recent/upcoming Netbox versions:
Sub-interfaces: https://github.com/netbox-community/netbox/issues/1519
Bridges: https://github.com/netbox-community/netbox/issues/6346
I think it makes total sense for us to leverage these built-in types once we upgrade to a supporting Netbox release.
Re-image cookbook changes
If we did use these types the re-image cookbook would need to be changed to:
- Get the primary IP interface as it is already
- If the interface has a connection then proceed as normal
- If the interface has no connection, get the interface type:
- If the interface "type" is a bridge.
- Cycle through the interfaces on device that belong to bridge
- If member device has a connection, get switch port from that
- Cycle through the interfaces on device that belong to bridge
- If the interface "type" is a bridge.
Which hopefully is not too hard to implement.
Puppet DB
A perhaps more difficult problem to solve is how to get the data into Netbox in the first place. This is added to Netbox by the puppetdb import script, however puppetdb does not define the interface types or relationships. For example:
cmooney@ganeti2010:~$ sudo facter -p networking
{
domain => "codfw.wmnet",
fqdn => "ganeti2010.codfw.wmnet",
hostname => "ganeti2010",
interfaces => {
eno1 => {
mac => "4c:d9:8f:6d:a0:85",
mtu => 1500
},
eno1.2003 => {
mac => "4c:d9:8f:6d:a0:85",
mtu => 1500
},
eno2 => {
mac => "4c:d9:8f:6d:a0:86",
mtu => 1500
},
lo => {
bindings => [
{
address => "127.0.0.1",
netmask => "255.0.0.0",
network => "127.0.0.0"
}
],
bindings6 => [
{
address => "::1",
netmask => "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff",
network => "::1"
}
],
ip => "127.0.0.1",
ip6 => "::1",
mtu => 65536,
netmask => "255.0.0.0",
netmask6 => "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff",
network => "127.0.0.0",
network6 => "::1"
},
private => {
bindings => [
{
address => "10.192.32.139",
netmask => "255.255.252.0",
network => "10.192.32.0"
}
],
bindings6 => [
{
address => "2620:0:860:103:4ed9:8fff:fe6d:a085",
netmask => "ffff:ffff:ffff:ffff::",
network => "2620:0:860:103::"
},
{
address => "fe80::4ed9:8fff:fe6d:a085",
netmask => "ffff:ffff:ffff:ffff::",
network => "fe80::"
}
],
ip => "10.192.32.139",
ip6 => "2620:0:860:103:4ed9:8fff:fe6d:a085",
mac => "4c:d9:8f:6d:a0:85",
mtu => 1500,
netmask => "255.255.252.0",
netmask6 => "ffff:ffff:ffff:ffff::",
network => "10.192.32.0",
network6 => "2620:0:860:103::"
},
public => {
bindings6 => [
{
address => "fe80::4ed9:8fff:fe6d:a085",
netmask => "ffff:ffff:ffff:ffff::",
network => "fe80::"
}
],
ip6 => "fe80::4ed9:8fff:fe6d:a085",
mac => "4c:d9:8f:6d:a0:85",
mtu => 1500,
netmask6 => "ffff:ffff:ffff:ffff::",
network6 => "fe80::"
}
},
ip => "10.192.32.139",
ip6 => "2620:0:860:103:4ed9:8fff:fe6d:a085",
mac => "4c:d9:8f:6d:a0:85",
mtu => 1500,
netmask => "255.255.252.0",
netmask6 => "ffff:ffff:ffff:ffff::",
network => "10.192.32.0",
network6 => "2620:0:860:103::",
primary => "private"
}The MAC address information might allow us to associate interfaces, but their exact types and which is parent/child would still be unavailable. MACs may not also be the same (although by default they would be).
Required Info
Ultimately we need to know what devices are bridges, and what other interfaces are members of each bridge:
cmooney@ganeti2010:~$ sudo brctl show bridge name bridge id STP enabled interfaces private 8000.4cd98f6da085 no eno1 public 8000.4cd98f6da085 no eno1.2003
As well as what 802.1q sub-interfaces are defined, and what their parent interfaces are:
cmooney@ganeti2010:~$ ip -d link show type vlan
5: eno1.2003@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master public state UP mode DEFAULT group default qlen 1000
link/ether 4c:d9:8f:6d:a0:85 brd ff:ff:ff:ff:ff:ff promiscuity 1
vlan protocol 802.1Q id 2003 <REORDER_HDR>This is where my own knowledge sort of runs out though. I'm not sure how or if we can get this information exposed in puppetdb, so it is available to the import script to populate Netbox correctly.
