While working on T289135 , I observed some IPMI errors emitted by the reimage cookbook .
Per conversation with Papaul in DC Ops IRC, the error means that IPMI is disabled on the node. We can potentially enable it in the idrac GUI under IPMI setting or with the provision script with the --no-dhcp --no-user flags .
Creating this ticket to address the issues, and find/add needed documentation.
Looks like the physical hosts installed in https://phabricator.wikimedia.org/T230746 ( elastic10[53-67].eqiad.wmnet ) never had IPMI enabled:
ansible -i ipmi.hosts --become all -m shell -a "ipmi-config --section=Lan_Channel --key-pair="Lan_Channel:Volatile_Access_Mode=Always_Available" --key-pair="Lan_Channel:Non_Volatile_Access_Mode=Always_Available" --diff" | tee ~/Documents/ansible-runs/ipmi-$(gdate -Iseconds).txt
Using these docs , enabled IPMI permissions for elastic10[53-67].eqiad.wmnet and ran the reimage playbook again. Leaving this ticket open until we confirm that the problem is fixed (keeping in mind that we may discover other hosts that need this fix).