ManagementSSHDown
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Sat, Jun 15, 10:47 PM

Description

Common information

dashboard: TODO
description: The management interface at db2197.mgmt:22 has been unresponsive for multiple hours.
runbook: https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
summary: Unresponsive management for db2197.mgmt:22

alertname: ManagementSSHDown
instance: db2197.mgmt:22
job: probes/mgmt
module: ssh_banner
prometheus: ops
rack: A2
severity: task
site: codfw
source: prometheus
team: dcops

Firing alerts

dashboard: TODO
description: The management interface at db2197.mgmt:22 has been unresponsive for multiple hours.
runbook: https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
summary: Unresponsive management for db2197.mgmt:22
alertname: ManagementSSHDown
instance: db2197.mgmt:22
job: probes/mgmt
module: ssh_banner
prometheus: ops
rack: A2
severity: task
site: codfw
source: prometheus
team: dcops
Source

Related Objects

Mentioned In: T368189: db2197 rebooted itself

Event Timeline

phaultfinder created this task.Sat, Jun 15, 10:47 PM

Restricted Application added a project: DC-Ops. · View Herald TranscriptSat, Jun 15, 10:47 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Maintenance_bot added a project: SRE.Sat, Jun 15, 11:29 PM

Papaul moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Sun, Jun 16, 6:54 PM

phaultfinder updated the task description. (Show Details)Mon, Jun 17, 12:32 AM

phaultfinder updated the task description. (Show Details)Tue, Jun 18, 1:17 PM

the iDrac light is rapidly blinking amber. I tried rebooting just the idrac by holding down the button for 45 seconds. it failed. The next troubleshooting step is to reboot and drain the power of the whole server. Please let me know when is a good time for this operation.

Removed both power supplies for 3 minutes and put them back. IDRAC is back up.

Marostegui mentioned this in T368189: db2197 rebooted itself.Sat, Jun 22, 12:29 PM

ManagementSSHDownClosed, ResolvedPublicActions