Page MenuHomePhabricator

hw troubleshooting: Can't reboot mw1349.eqiad.wmnet
Closed, ResolvedPublicRequest

Description

  • - Provide FQDN of system.
  • - If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.
  • - Put system into a failed state in Netbox.
  • - Provide urgency of request, along with justification (redundancy, dependencies, etc)
  • - Describe issue and/or attach hardware failure log. (Refer to https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook if you need help)
  • - Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

FQDN: mw1349.eqiad.wmnet

Following a soft reboot, server did not come back up.
Tried management interface powercycle and hardreset to no avail:

/admin1-> racadm serveraction powercycle

Unable to perform requested operation.
/admin1-> racadm
racadm>>serveraction powerstatus 

racadm serveraction powerstatus  
Server power status: ON

racadm>>serveraction powerdown 

racadm serveraction powerdown  
Unable to perform requested operation.

racadm>>serveraction hardreset 

racadm serveraction hardreset  
Unable to perform requested operation.

Host is depooled and part of a big cluster, urgency is medium.

Event Timeline

Server came back up as I pressed submit, however there still is an issue with the management interface. It does powercycle the server when asked, but states Unable to perform requested operation.

I'll keep the host depooled just in case.

You could try running "racadm racreset" over the serial console, it solved similar cases for me in the past. It will kick you off the current SSH connection to the mgmt, but you can usually reconnect after ~ 30 seconds.

Mentioned in SAL (#wikimedia-operations) [2023-09-06T14:22:31Z] <claime> Leaving mw1349.eqiad.wmnet pooled=invalid until management interface investigation - T345741

You could try running "racadm racreset" over the serial console, it solved similar cases for me in the past. It will kick you off the current SSH connection to the mgmt, but you can usually reconnect after ~ 30 seconds.

That solved it, thanks! Resolving then, sorry for the noise.

Mentioned in SAL (#wikimedia-operations) [2023-09-06T14:31:45Z] <claime> Repooling mw1349.eqiad.wmnet - T345741