elastic2020.codfw.wmnet is marked as down in icinga. By using the management console, I can check that the server is powered off (see log below). power on does not seem to work, the server still reports being powered off.
@Papaul I think this will require your expert hands.
gehel@durin:~$ ssh firstname.lastname@example.org email@example.com's password: User:root logged-in to ILOMXQ526080P.dasher.com(10.193.2.217 / FE80::EEB1:D7FF:FE78:2BBC) iLO 4 Advanced 2.20 at May 20 2015 Server Name: Server Power: Off Based on customer feedback, we will be enhancing the SSH command line interface in a future release of the iLO 4 firmware. Our future CLI will focus on increased usability and improved functionality. This message is to provide advance notice of the coming change. Please see the iLO 4 Release Notes on www.hp.com/go/iLO for additional information. </>hpiLO-> power off hard status=2 status_tag=COMMAND PROCESSING FAILED error_tag=COMMAND ERROR-UNSPECIFIED Mon Oct 24 12:09:46 2016 Server power already off. </>hpiLO-> power reset status=2 status_tag=COMMAND PROCESSING FAILED error_tag=COMMAND ERROR-UNSPECIFIED Mon Oct 24 12:09:54 2016 Server power off. </>hpiLO-> power on status=0 status_tag=COMMAND COMPLETED Mon Oct 24 12:10:03 2016 Server powering on ....... </>hpiLO-> vsp Virtual Serial Port Active: COM2 The server is not powered on. The Virtual Serial Port is not available. Starting virtual serial port. Press 'ESC (' to return to the CLI Session.