SystemdUnitFailed - gitlab2002
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Mar 11 2024, 6:39 PM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state

alertname: SystemdUnitFailed
instance: gitlab2002:9100
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: rsync-config-backup-gitlab1004.wikimedia.org.service on gitlab2002:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: rsync-config-backup-gitlab1004.wikimedia.org.service on gitlab2002:9100
alertname: SystemdUnitFailed
instance: gitlab2002:9100
name: rsync-config-backup-gitlab1004.wikimedia.org.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: rsync-data-backup-gitlab1004.wikimedia.org.service on gitlab2002:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: rsync-data-backup-gitlab1004.wikimedia.org.service on gitlab2002:9100
alertname: SystemdUnitFailed
instance: gitlab2002:9100
name: rsync-data-backup-gitlab1004.wikimedia.org.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

Related Objects

Mentioned Here: T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024

Event Timeline

phaultfinder created this task.Mar 11 2024, 6:39 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2024, 6:39 PM

phaultfinder updated the task description. (Show Details)Mar 11 2024, 8:59 PM

phaultfinder updated the task description. (Show Details)Mar 12 2024, 1:04 PM

related to T358559

Dzahn merged a task: T359961: SystemdUnitFailed.Mar 12 2024, 9:27 PM

Dzahn merged a task: T359995: SystemdUnitFailed.Mar 13 2024, 5:15 AM

Dzahn merged a task: T360011: SystemdUnitFailed.Mar 13 2024, 2:12 PM

This was related to the switchover between gitlab-replica and gitlab-replica-old. I've made improvements to the cookbook to leave a rollback in a better position.

SystemdUnitFailed - gitlab2002Closed, ResolvedPublicActions