NodeDown
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Dec 2 2023, 12:21 PM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1063
description: The cloudvirt node cloudvirt1063 is unreachable. This is a

major outage that requires you to either restore the server or evacuate manually the VMs on it.

runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown
summary: Cloudvirt node cloudvirt1063 is down. #page

alertname: NodeDown
cluster: wmcs
instance: cloudvirt1063:9100
job: node
prometheus: ops
severity: page
site: eqiad
source: prometheus
team: wmcs

Firing alerts

dashboard: https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1063
description: The cloudvirt node cloudvirt1063 is unreachable. This is a

major outage that requires you to either restore the server or evacuate manually the VMs on it.

runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown
summary: Cloudvirt node cloudvirt1063 is down. #page
alertname: NodeDown
cluster: wmcs
instance: cloudvirt1063:9100
job: node
prometheus: ops
severity: page
site: eqiad
source: prometheus
team: wmcs
Source

Related Objects
Search...

Status	Assigned	Task
Resolved	taavi	T352595 NodeDown
Resolved	Jclark-ctr	T353408 Cloudvirt1063.eqiad.wmnet overheating
Duplicate	None	T354491 NodeDown

Event Timeline

phaultfinder created this task.Dec 2 2023, 12:21 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 2 2023, 12:21 PM

-------------------------------------------------------------------------------
Record:      6
Date/Time:   12/02/2023 12:14:27
Source:      system
Severity:    Critical
Description: CPU 2 has a thermal trip (over-temperature) event.
-------------------------------------------------------------------------------

Mentioned in SAL (#wikimedia-cloud) [2023-12-02T12:27:35Z] <taavi> powercycle cloudvirt1063 T352595

taavi@cloudcontrol1006 ~ $ os server list --all --host cloudvirt1063
+--------------------------------------+-------------------------------+--------+----------------------------------------+----------------------------------------------+---------------------------------+
| ID                                   | Name                          | Status | Networks                               | Image                                        | Flavor                          |
+--------------------------------------+-------------------------------+--------+----------------------------------------+----------------------------------------------+---------------------------------+
| fcefcce4-c97d-44cf-a077-310612ed0118 | tools-k8s-worker-96           | ACTIVE | lan-flat-cloudinstances2b=172.16.6.246 | debian-10.0-buster                           | g3.cores8.ram16.disk20.ephem140 |
| 826760c6-d153-4070-9674-f1207c3ec328 | copypatrolbackenddeploytest01 | ACTIVE | lan-flat-cloudinstances2b=172.16.0.118 | debian-12.0-bookworm (deprecated 2023-11-27) | g3.cores2.ram4.disk20           |
| fd43cf27-3b19-4035-8047-7529c69b3e53 | project-proxy-acme-chief-02   | ACTIVE | lan-flat-cloudinstances2b=172.16.5.120 | debian-12.0-bookworm (deprecated 2023-11-27) | g3.cores1.ram2.disk20           |
| 81c76451-4919-4105-9142-410add2e4cfb | canary1063-1                  | ACTIVE | lan-flat-cloudinstances2b=172.16.2.170 | debian-11.0-bullseye                         | g3.cores1.ram1.disk20           |
+--------------------------------------+-------------------------------+--------+----------------------------------------+----------------------------------------------+---------------------------------+