This was discovered today by @Volans during T321311: msw-c5-eqiad offline. Namely some <host>.mgmt recovered even though the switch was still hard down. Turns out that bmc-config was returning 0.0.0.0 and thus ipmi_lan fact was set to that, which in turn meant icinga was updating its config to probe 0.0.0.0
Description
Description
Related Objects
Related Objects
- Mentioned Here
- T321311: msw-c5-eqiad offline
Event Timeline
Comment Actions
The icinga config for the mgmt hosts is generated by modules/monitoring/manifests/host.pp that has:
address => $facts['ipmi_lan']['ipaddress'],
and sudo facter -p 'ipmi_lan' was returning:
ipaddress => "0.0.0.0"
The ipmi_lan is a custom fact that comes from modules/ipmi/lib/facter/ipmi.rb and uses either /usr/sbin/bmc-config (if present) or /usr/bin/ipmitool (as a fallback) to get the data.
During the outage this is the output I got on ganeti1010:
$ sudo /usr/sbin/bmc-config -o -S Lan_Conf # # Section Lan_Conf Comments # # In the Lan_Conf section, typical networking configuration is setup. Most users # will choose to set "Static" for the "IP_Address_Source" and set the # appropriate "IP_Address", "MAC_Address", "Subnet_Mask", etc. for the machine. # Section Lan_Conf ## Possible values: Unspecified/Static/Use_DHCP/Use_BIOS/Use_Others IP_Address_Source Static ## Give valid IP address IP_Address 0.0.0.0 ## Give valid MAC address MAC_Address 4C:D9:8F:66:1D:D3 ## Give valid Subnet Mask Subnet_Mask 0.0.0.0 ## Give valid IP address Default_Gateway_IP_Address 0.0.0.0 ## Give valid MAC address Default_Gateway_MAC_Address 00:00:00:00:00:00 ## Give valid IP address Backup_Gateway_IP_Address 0.0.0.0 ## Give valid MAC address Backup_Gateway_MAC_Address 00:00:00:00:00:00 EndSection
The ipmitool tool is not installed there so I didn't try it.