Page MenuHomePhabricator

lawrencium's iDRAC misbehaving IPMI wise
Closed, ResolvedPublic

Description

I tried to reimage lawrencium today, but no dice.

10:42:56 | lawrencium.eqiad.wmnet | Unable to run wmf-auto-reimage-host: Remote IPMI failed for mgmt 'lawrencium.mgmt.eqiad.wmnet': Command '['ipmitool', '-I', 'lanplus', '-H', 'lawrencium.mgmt.eqiad.wmnet', '-U', 'root', '-E', 'chassis', 'power', 'status']' returned non-zero exit status 1

And

ipmitool -I lanplus -H lawrencium.mgmt.eqiad.wmnet -U root chassis status
Password: 
Error: Unable to establish IPMI v2 / RMCP+ session

I triple checked the pass, it's correct (it works against other hosts anyway)

Next was logging in via the iDRAC. That worked. A racreset did not solve the issue. I checked configuration and unless I am checking something really wrong it looks like iDRAC has IPMI enabled. cfgIpmiLanEnable=1 in the iDRAC's getconfig -g cfgIpmiLan command output.

Event Timeline

akosiaris claimed this task.
akosiaris added a subscriber: Volans.

With @Volans 's help we managed to fix it (the root cause is unknown).

Per T150160 a

IPMI passwords getting out of sync with their iDRAC passwords. An ssh root@$hostname racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i 2 $password fixes this usually

worked (after trying a few of the other workarounds in that task)