Page MenuHomePhabricator

elastic2018 not rebooting
Closed, ResolvedPublic

Description

elastic2018 was rebooted during normal maintenance, but never came back up.

console shows that it is stuck in the boot process, trying to boot over PXE and failing.

@Volans had a look at logs and found:

date=05/30/2018
time=23:32
description=POST Error: 266-Non-Volatile Memory Corruption Detected. Configuration settings restored to defaults. If enabled, Secure Boot security settings may be lost. Action: Restore desired configuration settings. Contact HP if issue persists.

Event Timeline

Gehel created this task.May 31 2018, 8:32 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 31 2018, 8:32 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-31T08:42:55Z] <gehel> power off elastic2018 - T196045

Having a look around in the system utility (ESC+9) I found that:

System Health Summary > System BIOS
> Health Status: Configuration Required

So it might be that somehow the BIOS config was lost during the reboot (or before?).
I had a quick look but I'm not fully sure of what are all the correct settings.

I'd recommend to have DC-Ops have a thorough check of all the BIOS options to make sure they are correct (not only boot order at this point, but all the other settings that we set on new installations too, cannot trust it at this point).

Gehel added a subscriber: Papaul.May 31 2018, 8:44 AM

@Papaul could you have a look at elastic2018 and see if you understand anything? The server is powered off, do anything you'd like with it...

MoritzMuehlenhoff triaged this task as Medium priority.
Papaul reassigned this task from Papaul to Gehel.May 31 2018, 3:00 PM

@Gehel for some reason, the server lost some settings like in the BIOS Serial Console & EMS EMS Console was COM1 , BOOT options was set to UEFI and USB options Internal SD Card Slot was enable. I changed back those settings the way it supposed to be. Power off and on the server 3 times using the ILO CLI . Server is back up

Gehel added a comment.May 31 2018, 5:07 PM

It looks like this worked, elastic2018 looks good again.

@Papaul is there any follow up we should do on that? Otherwise, feel free to close the task. Thanks!

Papaul closed this task as Resolved.May 31 2018, 5:30 PM

@Gehel no follow up at my end

Vvjjkkii renamed this task from elastic2018 not rebooting to mxbaaaaaaa.Jul 1 2018, 1:07 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed Gehel as the assignee of this task.
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from mxbaaaaaaa to elastic2018 not rebooting.Jul 2 2018, 2:12 AM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to Gehel.
CommunityTechBot lowered the priority of this task from High to Medium.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: Aklapper.