Page MenuHomePhabricator
Create Task
Maniphest T204567

ms-be2030 spontaneous reboot
Closed, ResolvedPublic
Actions

Description

ms-be2030 was completely unresponsive over the console this last weekend. I forced power cycle earlier this morning and it spontaneously rebooted about 9h later:

08:35 -icinga-wm:#wikimedia-operations- RECOVERY - Host ms-be2030 is UP: PING OK - Packet loss = 0%, RTA = 36.63 ms
17:08 -icinga-wm:#wikimedia-operations- PROBLEM - Host ms-be2030 is DOWN: PING CRITICAL - Packet loss = 100%
17:10 -icinga-wm:#wikimedia-operations- RECOVERY - Host ms-be2030 is UP: PING OK - Packet loss = 0%, RTA = 39.22 ms

Nothing from ipmi-sel though a spontaneous reboot is obviously troubling. @Papaul have you seen this before?

Event Timeline

fgiunchedi created this task.Sep 17 2018, 5:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2018, 5:04 PM
Papaul added a comment.Sep 17 2018, 6:13 PM

@fgiunchedi nerver seen this on HP servers. I also checked the log it just says "server reset" no other reason. Since the server is under warranty, I can open a case with HP and submit them the AHS logs they can provide me with more information after they review the log.
If they don't find anything, they will ask me o upgrade the server firmware.

Papaul added a comment.Sep 17 2018, 6:32 PM

@fgiunchedi I open a case with HP and for now They said there is no engineer available to help me, I will be receiving a call back in an hour. Please see below for case information.

Dear Mr Papaul,

Thank you for contacting Hewlett Packard Enterprise for your service request. This email confirms your request for service and the details are below.

Your request is being worked on under reference number 5332560753
Status: Case is generated and in Progress

Product description: HPE ProLiant DL380 Gen9 12LFF Configure-to-order Server
Product number: 719061-B21
Serial number: MXQ70601RV
Subject: DL380 Gen9 - Server rebooted - Fault analysis required

Papaul added a comment.Sep 17 2018, 8:43 PM

Engineer called at 2:20 PM CDT called ended at 2:37 PM CDT. He went over the AHS logs and didn't find any error or issue. His recommendation as I mentioned was to upgrade the BIOS and the controller. @fgiunchedi let me know.

Hello Sir ,
Greetings from HPE ,

As discussed over the call please refer to the below mentioned details

Link to BIOS : https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=1009087943&swItemId=MTX_ea86c71918094b7781bff4b124&swEnvOid=4184

BIOS is OS independent hen e do not panic to see the OS as Windows

You can directly afdate the BIOS from teh ILO COnsole

Download teh File
Extract it in a folder "BIOS 2018"
Go to ILO Home page >> Administrator >> Firmware >> Browse >> BIOS 2018 >> and select the largest file in the directory

Link to Controllers Firmware : https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=7274906&swItemId=MTX_94d23918a511422fa46c3f49f7&swEnvOid=4184

Controllers firmware is tested on RHEL hence you might face some issue while installing this , you can always give us a call back for any instant support
kindly refer to the installation instruction tab on the Link for your personal reference

Thanks & Regards,
Prakash Singh
Technical Solutions Consultant, HPE Enhanced Support
Hewlett Packard Enterprise | CSC, Bangalore
Email: prakash.tri.singh@hpe.com
You can reach our support staff at:

fgiunchedi added a comment.Sep 18 2018, 8:51 AM

@Papaul thanks! indeed looks like the server just reset. I'm skeptical upgrading bios/controller will help but no harm in trying either, let me know when you are online later today and I'll shut the host.

MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.Sep 18 2018, 10:54 AM

Server went down again at 10:45 UTC.

Volans added a subscriber: Volans.Sep 18 2018, 10:57 AM

Console unresponsive, nothing in show /system1/log1/, no ping. As agreed on IRC I'm leaving it down for now until @fgiunchedi comes back from lunch to allow further investigation.

fgiunchedi added a comment.Sep 18 2018, 12:11 PM

Also power commands from ilo don't seem to respond timely or work at all:

</>hpiLO-> power off
                    
status=0
status_tag=COMMAND COMPLETED
Tue Sep 18 12:09:16 2018
                        


Server powering off .......



</>hpiLO-> power
                
status=0
status_tag=COMMAND COMPLETED
Tue Sep 18 12:09:19 2018
                        


power: server power is currently: On


</>hpiLO-> power
                
status=0
status_tag=COMMAND COMPLETED
Tue Sep 18 12:09:33 2018
                        


power: server power is currently: Resetting


</>hpiLO-> power
                
status=0
status_tag=COMMAND COMPLETED
Tue Sep 18 12:09:43 2018
                        


power: server power is currently: Resetting

</>hpiLO-> power
                
status=0
status_tag=COMMAND COMPLETED
Tue Sep 18 12:10:28 2018
                        


power: server power is currently: On
fgiunchedi added a comment.Sep 18 2018, 12:23 PM

The host is back up now, clearly not stable enough for production though.

Stashbot added a subscriber: Stashbot.Sep 18 2018, 1:56 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-18T13:56:30Z] <godog> poweroff ms-be2030 - T204567

Papaul added a comment.Sep 18 2018, 2:36 PM

@fgiunchedi getting "The last firmware update attempt was not successful. Ready for the next update." error when trying to update the BIOS. so I email the HP engineer and waiting for response.

Since using the ILO to upgrade the BIOS is not working, I will have to make a new Service Pack disk to upgrade the firmware on the server. the SP disk that i have on site right now is old.

Papaul added a comment.Sep 19 2018, 5:31 PM

@fgiunchedi HW diagnostics didn't report any problem. Upgrade all firmware on the server . we can observe the server all this week and if we do have the same problem again, I will contact HP.

Papaul added a comment.Sep 20 2018, 10:47 PM

The server reset again. This time I have something in the log file. see below

Selection_038.png (104×1 px, 18 KB)

Papaul added a comment.Sep 20 2018, 11:30 PM

Called HP at 17:48 CDT call ended at 18:25 CDT. went over again the new Log from today and didn't see any internal problem on the server. The only thing he asked me to do it to change the power settings of the server from Dynamic power saving mode to Static High performance mode. The error reported on the integrated management log according (Critical Temperature Threshold Exceeded) to him is an external issue.

MoritzMuehlenhoff assigned this task to Papaul.Sep 24 2018, 10:07 AM
MoritzMuehlenhoff triaged this task as Medium priority.
Papaul reassigned this task from Papaul to fgiunchedi.Sep 25 2018, 1:46 PM

@fgiunchedi Since 9-20-18 after making the power settings change, I have been monitoring the server; so far the server has been up with no reset. Is it possible to put back the server in production so it can get some load and I will be monitoring it and report back to HP.

Thanks

fgiunchedi reassigned this task from fgiunchedi to Papaul.Sep 25 2018, 1:55 PM

Thanks @Papaul, the server has been in production the whole time and indeed no power resets so far.

Papaul added a comment.Sep 25 2018, 2:59 PM

@fgiunchedi thank you will leave this task open until the end of the week.

Papaul closed this task as Resolved.Oct 1 2018, 2:50 PM

I checked again the server logs, everything looks good. Resolving this task.

MoritzMuehlenhoff reopened this task as Open.Feb 4 2019, 2:22 PM

I'm reopening the task, the server went down again today:

[13:56] <icinga-wm> PROBLEM - Host ms-be2030 is DOWN: PING CRITICAL - Packet loss = 100%

I was looking at system logs and while I was doing that the server came back up:

[14:12] <icinga-wm> RECOVERY - Host ms-be2030 is UP: PING OK - Packet loss = 0%, RTA = 37.99 ms

The system logs indicate a heating error:

/system1/log1/record16
  Targets
  Properties
    number=16
    severity=Critical
    date=02/04/2019
    time=14:03
    description=Critical Temperature Threshold Exceeded
                
/system1/log1/record17
  Targets
  Properties
    number=17
    severity=Caution
    date=02/04/2019
    time=14:11
    description=Option ROM POST Error: 1719-Slot 3 Drive Array - A controller failure event occurred prior to this power-up. (Previous lock up code = 0xAB) Action: Install the latest controller firmware. If the problem persists, replace the controller.
Papaul added a comment.Feb 8 2019, 5:49 PM

Checked temperature in the rack all looks good. add blanks to the rack since we have only 8 servers in that rack. Leaving the task open for another week.

Papaul added a comment.Feb 21 2019, 4:54 PM

@fgiunchedi is it possible to depool this server for me to do a firmware upgrade before I resolve the task?

fgiunchedi added a comment.Feb 22 2019, 9:51 AM
In T204567#4972487, @Papaul wrote:

@fgiunchedi is it possible to depool this server for me to do a firmware upgrade before I resolve the task?

Yes, a clean shutdown of the host is enough, let me know later today when you are online and I'll do the poweroff.

Papaul added a comment.Feb 25 2019, 4:05 PM

@fgiunchedi we can do this tomorrow if thats okay with you. Thanks.

fgiunchedi added a comment.Feb 26 2019, 9:04 AM
In T204567#4981395, @Papaul wrote:

@fgiunchedi we can do this tomorrow if thats okay with you. Thanks.

Sounds good to me -- ping me on IRC when good to go!

Stashbot added a comment.Feb 26 2019, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-26T15:13:48Z] <godog> poweroff ms-be2030 - T204567

Papaul closed this task as Resolved.Feb 26 2019, 4:10 PM

Firmware upgrade complete. Resolving this task for now.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL