KernelErrors Server cloudcephosd1013 logged kernel errors
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jul 12 2025, 4:24 PM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephosd1013
description: The server cloudcephosd1013 logged some kernel errors in the past 24h
runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors
summary: Server cloudcephosd1013 logged kernel errors

alertname: KernelErrors
category: priority_err
cluster: wmcs
instance: cloudcephosd1013:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs

Firing alerts

dashboard: https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephosd1013
description: The server cloudcephosd1013 logged some kernel errors in the past 24h
runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors
summary: Server cloudcephosd1013 logged kernel errors
alertname: KernelErrors
category: priority_err
cluster: wmcs
instance: cloudcephosd1013:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Request	Jclark-ctr	T401319 hw troubleshooting: disk sdj failure for cloudcephosd1013.eqiad.wmnet
		Resolved		fnegri	T399366 KernelErrors Server cloudcephosd1013 logged kernel errors

Event Timeline

phaultfinder created this task.Jul 12 2025, 4:24 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 12 2025, 4:24 PM

This was due to a reboot, after sdj disappeared.

This was due to a reboot, after sdj disappeared.

The alert was not due to the reboot. The metric was reporting 318 kernel errors starting on Jul 11 at 14:17 UTC (before the reboot). The reboot happened on Jul 11 at 18:36 UTC. The alert did not fire until Jul 12 only because there was a silence in place that expired on Jul 12.

Screenshot 2025-07-14 at 17.37.13.png (1×2 px, 368 KB)

A sample of the 318 kernel errors:

Jul 11 14:12:32 cloudcephosd1013 kernel: INFO: task md2_raid1:668 blocked for more than 120 seconds.
Jul 11 14:12:32 cloudcephosd1013 kernel:       Not tainted 5.10.0-35-amd64 #1 Debian 5.10.237-1
Jul 11 14:12:32 cloudcephosd1013 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[...]

Jul 11 14:12:32 cloudcephosd1013 kernel: megaraid_sas 0000:18:00.0: pending commands remain after waiting, will reset adapter scsi0.
Jul 11 14:12:32 cloudcephosd1013 kernel: blk_update_request: I/O error, dev sdj, sector 427105432 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
Jul 11 14:12:32 cloudcephosd1013 kernel: scsi 0:0:8:0: rejecting I/O to dead device

[...]

Jul 11 14:12:32 cloudcephosd1013 kernel: Buffer I/O error on dev dm-4, logical block 226257573, async page read

fnegri mentioned this in T399360: KernelErrors .Jul 16 2025, 4:19 PM

fnegri triaged this task as Medium priority.Jul 23 2025, 2:07 PM

Follow-up: I asked DCops to take out the failed drive in T401319: hw troubleshooting: disk sdj failure for cloudcephosd1013.eqiad.wmnet.

fnegri mentioned this in T404300: Remove KernelErrors alerts.Sep 11 2025, 9:04 AM

	F64382207: Screenshot 2025-07-14 at 17.37.13.png
	Jul 14 2025, 3:45 PM

KernelErrors Server cloudcephosd1013 logged kernel errorsClosed, ResolvedPublicActions