Leaving this documented for the future. I tried a cold reset locally, but it doesn't fix the remote issue.

root@db1070:~# bmc-device --debug --cold-reset
=====================================================
Cold Reset Request
=====================================================
[               2h] = cmd[ 8b]
=====================================================
Cold Reset Response
=====================================================
[               2h] = cmd[ 8b]
[               0h] = comp_code[ 8b]
root@db1070:~# bmc-device --get-sel-time
ipmi_cmd_get_sel_time: BMC busy
root@db1070:~# bmc-device --get-sel-time
SEL Time : 04/26/2017 - 14:53:12

But that doesn't fixed the remote problem:

root@neodymium:/home/marostegui/git/software/dbtools# ipmitool -I lanplus -H db1070.mgmt.eqiad.wmnet -U root -E chassis
Unable to read password from environment
Password:
Error: Unable to establish IPMI v2 / RMCP+ session

@Marostegui I am not sure what to make of this...i know several servers have this issue but on the server itself ipmi works fine.

@Cmjohnson this looks similar as the same issue with lots of servers, including dbstore1001 on this task: T158893#3186029 if we get Dell to fix this or advise on that ticket, probably we can apply the same for this host (and other ones suffering from the same issue).

Change 351822 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1070

https://gerrit.wikimedia.org/r/351822

Change 351822 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1070

https://gerrit.wikimedia.org/r/351822

Mentioned in SAL (#wikimedia-operations) [2017-05-04T12:40:22Z] <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Depool db1070 for maintenance - T160392 (duration: 01m 35s)

Mentioned in SAL (#wikimedia-operations) [2017-05-04T12:42:41Z] <marostegui> Stop MySQL db1070 for maintenance - T160392

Mentioned in SAL (#wikimedia-operations) [2017-05-04T14:09:44Z] <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Repool db1070 with less weight - T160392 (duration: 01m 16s)

Mentioned in SAL (#wikimedia-operations) [2017-05-04T14:24:05Z] <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Increase db1070 weight - T160392 (duration: 01m 10s)

Mentioned in SAL (#wikimedia-operations) [2017-05-04T15:03:24Z] <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Restore db1070 original weight - T160392 (duration: 00m 57s)

@Cmjohnson just checking if in the end you updated the idrac firmware? No pushing by any means, just checking if I need to powercycle this host next week or not.
Thank you!

Change 352171 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] Depool db1070 for hardware maintenance

https://gerrit.wikimedia.org/r/352171

Change 352171 merged by jenkins-bot:
[operations/mediawiki-config@master] Depool db1070 for hardware maintenance

https://gerrit.wikimedia.org/r/352171

Mentioned in SAL (#wikimedia-operations) [2017-05-05T16:09:57Z] <jynus> shutting down db1070 for hw maintenance T160392

I updated the firmware on db1070 and ipmitool is still not working, I compared the idrac settings via the gui with db1068 (ipmi works) and not differences between the two that I could tell. I even tried swapping the cable to see if it could be that (longshot). Still no idea why ipmi is not working. All Dell troubleshooting requires several hours of trial and error and booting in to their Dell live image. I will need to schedule extended downtime with this or dbstore1001 or db1071 with @Marostegui or @jcrespo.

I did update the bios f/w while it was down.

Thanks @Cmjohnson for all the help.
We can test with db1070 for as much as you like during the week (I just don't like leaving hosts down for the weekend) just let me know when you need it down and I will handle it.
The only reason not to choose dbstore1001 is because it is our backups server, but if we find the issue with db1070, it is likely that dbstore (and the rest of affected hosts) are suffering the same.

Thanks for spending time on this, we appreciate it!

FYI, db1071 is in a similar state, I'm not sure why.

Nice catch faidon!! Thanks for fixing this and specially thanks for fixing dbstore1001, which is a critical host for us!