NodeDown cloudvirt1063
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Dec 14 2023, 12:20 AM

Description

dashboard: https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1063
description: The cloudvirt node cloudvirt1063 is unreachable. This is a

major outage that requires you to either restore the server or evacuate manually the VMs on it.

runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown
summary: Cloudvirt node cloudvirt1063 is down. #page

dashboard: https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1063
description: The cloudvirt node cloudvirt1063 is unreachable. This is a

major outage that requires you to either restore the server or evacuate manually the VMs on it.

runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown
summary: Cloudvirt node cloudvirt1063 is down. #page
alertname: NodeDown
cluster: wmcs
instance: cloudvirt1063:9100
job: node
prometheus: ops
severity: page
site: eqiad
source: prometheus
team: wmcs
Source

Status	Assigned	Task
		Unknown Object (Task)
Resolved	Papaul	T342537 Q1:rack/setup/install cloudvirt10[62-67]
Resolved	fnegri	T353406 NodeDown cloudvirt1063
Resolved	Jclark-ctr	T353408 Cloudvirt1063.eqiad.wmnet overheating
Duplicate	None	T354491 NodeDown

Mentioned in SAL (#wikimedia-cloud) [2023-12-14T00:25:02Z] <andrewbogott> evacuating hosts from cloudvirt1063 and depooling. T353406

fnegri renamed this task from NodeDown to NodeDown cloudvirt1063.Dec 14 2023, 9:52 AM

Mentioned in SAL (#wikimedia-cloud) [2023-12-21T16:51:58Z] <dhinus> puppet node deactivate cloudvirt1063.eqiad.wmnet T353406

I deactivated the node to resolve the alerts until the server is replaced in T353408: Cloudvirt1063.eqiad.wmnet overheating