Maniphest T393345

SystemdUnitFailed - vrts1003 - user-runtime-dir@499
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	May 5 2025, 1:19 AM

Tags

Referenced Files

None

Subscribers

Description

Common information

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state

alertname: SystemdUnitFailed
instance: vrts1003:9100
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: user-runtime-dir@499.service on vrts1003:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: user-runtime-dir@499.service on vrts1003:9100
alertname: SystemdUnitFailed
instance: vrts1003:9100
name: user-runtime-dir@499.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: user@499.service on vrts1003:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: user@499.service on vrts1003:9100
alertname: SystemdUnitFailed
instance: vrts1003:9100
name: user@499.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

Related Objects

Mentioned Here: T393366: Regression in RAID10 software RAID with 6.1.135
T393357: soft lockup on prometheus, centrallog, vrts hosts with the new kernel

Event Timeline

phaultfinder created this task.May 5 2025, 1:19 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 5 2025, 1:19 AM

phaultfinder updated the task description. (Show Details)May 5 2025, 1:29 AM

LSobanski renamed this task from SystemdUnitFailed to SystemdUnitFailed - vrts1003 - user-runtime-dir@499.May 5 2025, 8:29 AM

There's also an MD RAID alert for both vrts hosts: https://alerts.wikimedia.org/?q=alertname%3DMD%20RAID&q=team%3Dsre&q=%40receiver%3Dirc-spam

And an SSH socket timeout: https://alerts.wikimedia.org/?q=alertname%3DSSH&q=team%3Dsre&q=%40receiver%3Dirc-spam

There is a gap in host metrics between 2025-05-05 01:24 and 08:11 and then another one between 09:03 and 09:06.

Likely related to T393357: soft lockup on prometheus, centrallog, vrts hosts with the new kernel

Both vrts hosts were rebooted.

LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.May 5 2025, 10:38 AM

Addressed in T393366: Regression in RAID10 software RAID with 6.1.135