is it okay to power off or do these need to be scheduled?

mw1196 did not come back, fans were running but no output on crash cart. downed it but probably needs decom

In addition to these, the following return 192.168.0.0/16 addresses for either their address or the gateway:

• analytics1047.eqiad.wmnet
• analytics1061.eqiad.wmnet
• auth2001.codfw.wmnet
• cp1045.eqiad.wmnet
• cp2010.codfw.wmnet
• db2074.codfw.wmnet
• ganeti1006.eqiad.wmnet
• logstash1005.eqiad.wmnet
• mw1230.eqiad.wmnet
• mw2172.codfw.wmnet
• mw2204.codfw.wmnet
• ores1008.eqiad.wmnet
• pc2006.codfw.wmnet
• stat1004.eqiad.wmnet

These may be fixed by an iDRAC/iLO reset (racadm racreset or cd /map1, reset respectively).

Regarding sodium it seems to me that puppet runs get stuck because it try to execute ipmi-config while loading facts:

143053 pts/0    S+     0:00  |                       \_ /bin/bash /usr/local/sbin/run-puppet-agent
143080 pts/0    Sl+    0:02  |                           \_ /usr/bin/ruby /usr/bin/puppet agent --onetime --no-daemonize --verbose --no-splay --show_diff --ignorecache --no-usecacheonfailure
143331 pts/0    S+     0:29  |                               \_ /usr/sbin/ipmi-config --category=core -o -S Lan_Conf

It eventually get unblocked after 10 minutes printing Unable to get Number of Users and the puppet run continues.

# time run-puppet-agent
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Unable to get Number of Users
Info: Caching catalog for sodium.wikimedia.org
Info: Applying configuration version '1499726057'
Notice: /Stage[main]/Mirrors::Serve/Letsencrypt::Cert::Integrated[mirrors]/Exec[acme-setup-acme-mirrors]/returns: executed successfully
Notice: Finished catalog run in 10.25 seconds

real	10m24.171s
user	0m46.704s
sys	0m7.784s

analytics1047 already seemed ok, showed the right IP, racreset anyways but it stayed the same

analytics1061 also showed the right IP but the wrong gateway. (192.168.0.1 instead of 10.65.0.1 with IP 10.65.4.99) but it was fixed after a racreset.

I racreset all of the ones in list which had a discrepancy of their IP configuration with the output (showing 192.168.0.1 as gateway) and they're all fixed now.

I tried to racreset all ones in the "somewhat responsive" list and actually made them worse :) It seems that racreset from either bmc-device, SSH (even racadm racreset hard -f) or the web interface is not actually resetting properly. bmc-config stopped working but the BMC never reset properly (e.g. my SSH session remained active). These will need a proper power reset (thankfully they're just mw* hosts).

Mentioned in SAL (#wikimedia-operations) [2017-07-11T17:20:45Z] <mutante> mw2201, mw2202 - depool appservers for T169360 (drain flea power)

on both mw2201 and mw2202 I am getting

I can not reset the IDRAC in the BIOS also.
This looks like HW problem will have to contact Dell for main board replacement.

worked with papaul to drain flea power for the remaining codfw ones:

mw2154 has been fixed after draining flea power and now works
mw2201/mw2202 did not get fixed by it, see papaul's comment above - creating subtask to contact Dell and replace board

Mentioned in SAL (#wikimedia-operations) [2017-07-11T19:17:58Z] <paravoid> shutting down sodium for iDRAC reset (T169360)

mw2201 is fixed now.

mw2202 is fixed as well.