Maniphest T352570

CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Dec 1 2023, 7:16 PM

Tags

Referenced Files

None

Subscribers

Description

Common information

dashboard: https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS
description: Ceph cluster in eqiad has slow ops, which might be blocking some writes
runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps
summary: Ceph cluster in eqiad has 6 slow ops

alertname: CephSlowOps
cluster: wmcs
job: ceph_eqiad
prometheus: cloud
service: ceph,cloudvps
severity: critical
site: eqiad
source: prometheus
team: wmcs

Firing alerts

dashboard: https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS
description: Ceph cluster in eqiad has slow ops, which might be blocking some writes
runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps
summary: Ceph cluster in eqiad has 6 slow ops
alertname: CephSlowOps
cluster: wmcs
job: ceph_eqiad
prometheus: cloud
service: ceph,cloudvps
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		dcaro	T334240 [cloudceph] Slow operations - tracking task
		Resolved		taavi	T348634 ceph slow ops 2023-10-11
					Unknown Object (Task)
					Unknown Object (Task)
		In Progress		dcaro	T348643 cloudcephosd1021-1034: hard drive sector errors increasing
		Resolved		taavi	T352570 CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes

Event Timeline

phaultfinder created this task.Dec 1 2023, 7:16 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 1 2023, 7:16 PM

phaultfinder updated the task description. (Show Details)Dec 1 2023, 8:58 PM

phaultfinder updated the task description. (Show Details)Dec 5 2023, 3:04 AM

phaultfinder updated the task description. (Show Details)Dec 5 2023, 3:10 AM

phaultfinder updated the task description. (Show Details)Dec 5 2023, 7:28 PM

phaultfinder updated the task description. (Show Details)Dec 6 2023, 5:25 PM

dcaro added a parent task: T348643: cloudcephosd1021-1034: hard drive sector errors increasing.Dec 7 2023, 11:03 AM

phaultfinder updated the task description. (Show Details)Dec 9 2023, 8:53 AM

phaultfinder updated the task description. (Show Details)Dec 10 2023, 8:57 AM

phaultfinder updated the task description. (Show Details)Dec 11 2023, 8:00 AM

phaultfinder updated the task description. (Show Details)Dec 11 2023, 11:25 AM

phaultfinder updated the task description. (Show Details)Dec 12 2023, 3:15 AM

phaultfinder updated the task description. (Show Details)Dec 13 2023, 3:31 PM

phaultfinder updated the task description. (Show Details)Dec 14 2023, 11:36 AM

phaultfinder updated the task description. (Show Details)Dec 15 2023, 12:08 PM

phaultfinder updated the task description. (Show Details)Dec 16 2023, 11:13 AM

phaultfinder updated the task description. (Show Details)Dec 17 2023, 10:07 AM

phaultfinder updated the task description. (Show Details)Dec 18 2023, 3:01 PM

phaultfinder updated the task description. (Show Details)Dec 18 2023, 8:54 PM

phaultfinder updated the task description. (Show Details)Dec 19 2023, 3:33 AM

phaultfinder updated the task description. (Show Details)Dec 19 2023, 6:55 PM

phaultfinder updated the task description. (Show Details)Dec 20 2023, 3:15 PM

phaultfinder updated the task description. (Show Details)Dec 25 2023, 12:02 PM

phaultfinder updated the task description. (Show Details)Jan 4 2024, 12:35 PM

phaultfinder updated the task description. (Show Details)Jan 14 2024, 4:06 PM

phaultfinder updated the task description. (Show Details)Jan 15 2024, 11:53 AM

phaultfinder updated the task description. (Show Details)Jan 29 2024, 11:42 AM

phaultfinder updated the task description. (Show Details)Feb 5 2024, 11:39 AM

phaultfinder updated the task description. (Show Details)Feb 7 2024, 3:37 PM

phaultfinder updated the task description. (Show Details)Feb 8 2024, 6:32 PM

taavi closed this task as Resolved.Feb 27 2024, 8:38 AM

taavi claimed this task.