As part of the issues yesterday, we started looking into monitoring our iDRAC/iLOs (also see T169321). I deployed a Puppet fact today that fetches the IP/MAC address etc. from the BMCs. The fact basically runs `bmc-config -o -S Lan_Conf`. As part of that, I found various BMCs (mostly Dells) in weird states and was able to fix some of them. These states were:
- Returning stale IP addresses (fixed with a `racadm racreset`)
- Returning the IP address, then hanging for a while and never returning MAC address/gateway/netmask
- Not returning the MAC address
- Unresponsive from within the machine, but responsive from the network. In some cases fixed with `racadm racreset`, in others not responding at all
- Completely unresponsive from both within the machine, as well as externally.
For the ones I didn't manage to fix, we'll need to put out of commission, drain flea power and power on again. If that doesn't fix it, we should do an iDRAC firmware upgrade as well (or perhaps we should do it regardless, if it's easy).
**Servers with unresponsive iDRACs:**
[x] db1053.eqiad.wmnet
[x] sodium.wikimedia.org
[x] mw1182.eqiad.wmnet
[x] mw1190.eqiad.wmnet
[x] mw1191.eqiad.wmnet
[ ] mw1196.eqiad.wmnet
[ ] mw1199.eqiad.wmnet
[x] mw2154.codfw.wmnet
[ ] mw2201.codfw.wmnet
[ ] mw2202.codfw.wmnet
[x] labsdb1001.eqiad.wmnet (Cisco, ignore)
[x] labsdb1003.eqiad.wmnet (Cisco, ignore)
**Servers with responsive iDRAC/iLO, returning wrong LAN information (different from what configured), can probably be fixed with a BMC reset:**
[x] analytics1047.eqiad.wmnet
[x] analytics1061.eqiad.wmnet
[x] cp1045.eqiad.wmnet
[x] ganeti1006.eqiad.wmnet
[x] logstash1005.eqiad.wmnet
[x] mw1230.eqiad.wmnet
[x] ores1008.eqiad.wmnet
[x] stat1004.eqiad.wmnet
[x] auth2001.codfw.wmnet
[x] db2074.codfw.wmnet
[x] cp2010.codfw.wmnet
[x] mw2172.codfw.wmnet
[x] mw2204.codfw.wmnet
[x] pc2006.codfw.wmnet