ipmi calls seem to fail when trying to reimage. Either the service is disabled, degraded or there is some other configuration error. Needs checking. Regular management inferface works as intended, but it forced a manual reimage.
We should check logical and network configuration, then try a "power drain" to get it to respond.
Now that I have a way to test it, we can proceed, depooling:
$ ipmitool -I lanplus -H es2013.mgmt.codfw.wmnet -U root -E chassis power status Unable to read password from environment Password: Error: Unable to establish IPMI v2 / RMCP+ session $ ipmitool -I lanplus -H es2012.mgmt.codfw.wmnet -U root -E chassis power status Unable to read password from environment Password: Chassis Power is on $ ipmitool -I lanplus -H es2011.mgmt.codfw.wmnet -U root -E chassis power status Unable to read password from environment Password: Chassis Power is on
Change 425835 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool es2013
Change 425835 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Depool es2013
@Papaul you are now free to handle the server- it is up, but with all the service down and depooled. I would try the reset I proposed earlier first, and if that doesn't work, checking bios/admin config, maybe?
1- Power drain
2- Reset IDRAC
3- Update BIOS from 2.1.7 to 2.7.1
4- Update IDRAC from 2.21 to 2.52
It is still not working:
root@neodymium:/home/marostegui# ipmitool -I lanplus -H es2013.mgmt.codfw.wmnet -U root -E chassis power status Unable to read password from environment Password: Error: Unable to establish IPMI v2 / RMCP+ session
Mentioned in SAL (#wikimedia-operations) [2018-04-12T16:07:20Z] <marostegui> Reboot es2013 - T191977
I've fixed it, it was a case of password misalignment, see one of the cases described in T150160,