KernelErrors
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jul 12 2025, 12:24 PM

Description

Common information

runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors

alertname: KernelErrors
cluster: wmcs
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs

Firing alerts

dashboard: https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephosd1013
description: The server cloudcephosd1013 logged some kernel errors in the past 24h
runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors
summary: Server cloudcephosd1013 logged kernel errors
alertname: KernelErrors
category: keyword_taint
cluster: wmcs
instance: cloudcephosd1013:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephosd1013
description: The server cloudcephosd1013 logged some kernel errors in the past 24h
runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors
summary: Server cloudcephosd1013 logged kernel errors
alertname: KernelErrors
category: priority_err
cluster: wmcs
instance: cloudcephosd1013:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephosd1036
description: The server cloudcephosd1036 logged some kernel errors in the past 24h
runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors
summary: Server cloudcephosd1036 logged kernel errors
alertname: KernelErrors
category: keyword_taint
cluster: wmcs
instance: cloudcephosd1036:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephosd1036
description: The server cloudcephosd1036 logged some kernel errors in the past 24h
runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors
summary: Server cloudcephosd1036 logged kernel errors
alertname: KernelErrors
category: priority_err
cluster: wmcs
instance: cloudcephosd1036:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

Related Objects
Search...

Status	Assigned	Task
Resolved	Andrew	T399858 Cloud Ceph misbehaving on Debian Bookworm
Resolved	taavi	T399870 [k8s,infra,o11y] Add paging alert when many tools are unreachable
Resolved	Andrew	T399281 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures
Resolved	fnegri	T399360 KernelErrors

Event Timeline

phaultfinder created this task.Jul 12 2025, 12:24 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 12 2025, 12:24 PM

cloudcephosd1013 had a hard drive failure, see T399366: KernelErrors Server cloudcephosd1013 logged kernel errors.

cloudcephosd1036 had a single error message logged on 2025-07-11 15:56 UTC during the outage (T399281: 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures). Unfortunately the journald logs have already been rotated and I cannot see that message anymore. It was one of the hosts that were upgraded to Bookworm and were struggling, but it's the only one who reported a Kernel error.

Resolving as I don't think there's much more investigation we can do.

fnegri added a parent task: T399281: 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures.Jul 16 2025, 4:19 PM

KernelErrors Closed, ResolvedPublicActions