Page MenuHomePhabricator

hw troubleshooting: server fails to reboot for clouddb1018.eqiad.wmnet
Closed, ResolvedPublicRequest

Description

I rebooted clouddb1018.eqiad.wmnet with

fnegri@cumin1002:~$ sudo cookbook sre.hosts.reboot-single -r T366555 clouddb1018.eqiad.wmnet

The host failed to restart. console com2 is blank. I tried a few resets to no avail:

serveraction powercycle
serveraction hardreset
racreset

The server is currently depooled, and it can remain depooled for a few days.

  • - Provide FQDN of system.
  • - If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.
  • - Put system into a failed state in Netbox.
  • - Provide urgency of request, along with justification (redundancy, dependencies, etc)
  • - Describe issue and/or attach hardware failure log. (Refer to https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook if you need help)
  • - Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2024-06-14T12:08:57Z] <fnegri@cumin1002> START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on clouddb1018.eqiad.wmnet with reason: hardware issues T367499

Mentioned in SAL (#wikimedia-operations) [2024-06-14T12:09:01Z] <fnegri@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on clouddb1018.eqiad.wmnet with reason: hardware issues T367499

Sometimes it means the host is stuck at a memory check - should be visible onsite.

@Marostegui Updated idrac and bios firmware

took server down to min configuration and slowly added memory till all was reinstalled

Server is back up now