Maniphest T365781

SystemdUnitFailed - gitlab1003, gitlab1004, lists2001
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	May 24 2024, 4:09 AM

Tags

Referenced Files

None

Subscribers

Description

Common information

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state

alertname: SystemdUnitFailed
prometheus: ops
severity: critical
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: backup-restore.service on gitlab1003:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: backup-restore.service on gitlab1003:9100
alertname: SystemdUnitFailed
instance: gitlab1003:9100
name: backup-restore.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: backup-restore.service on gitlab1004:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: backup-restore.service on gitlab1004:9100
alertname: SystemdUnitFailed
instance: gitlab1004:9100
name: backup-restore.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: discard_held_messages.service on lists2001:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: discard_held_messages.service on lists2001:9100
alertname: SystemdUnitFailed
instance: lists2001:9100
name: discard_held_messages.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: stewards_subscriber_data_sync.service on lists2001:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: stewards_subscriber_data_sync.service on lists2001:9100
alertname: SystemdUnitFailed
instance: lists2001:9100
name: stewards_subscriber_data_sync.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T291916 Tracking task for Bullseye migrations in production
		Resolved		Dzahn	T327068 Bullseye upgrade for remaining Collab hosts
		Resolved		• eoghan	T331706 Migrate Mailman/lists to Bullseye/Bookworm
		Resolved		• eoghan	T283615 Make mailman3 work in the standby host (lists2001.wikimedia.org)
		Resolved		• eoghan	T365781 SystemdUnitFailed - gitlab1003, gitlab1004, lists2001

Event Timeline

phaultfinder created this task.May 24 2024, 4:09 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 24 2024, 4:09 AM

Jelto renamed this task from SystemdUnitFailed to SystemdUnitFailed (gitlab1003).May 24 2024, 7:27 AM

The GitLab alerts happened because of the GitLab upgrade in T365587 and the current version mismatch.

@eoghan I'll rename the task because of the lists alerts

Jelto renamed this task from SystemdUnitFailed (gitlab1003) to SystemdUnitFailed (lists2001).May 24 2024, 7:30 AM

Jelto merged a task: T365722: SystemdUnitFailed - backup-restore.service - gitlab1003.

Jelto added a parent task: T331706: Migrate Mailman/lists to Bullseye/Bookworm.

Jelto edited parent tasks, added: T283615: Make mailman3 work in the standby host (lists2001.wikimedia.org); removed: T331706: Migrate Mailman/lists to Bullseye/Bookworm.May 24 2024, 7:32 AM

LSobanski renamed this task from SystemdUnitFailed (lists2001) to SystemdUnitFailed - gitlab1003, gitlab1004, lists2001.May 27 2024, 5:58 AM

The lists alert was me deploying puppet changes, with more details in T365698

• eoghan mentioned this in T365789: SystemdUnitFailed - gitlab1003, gitlab1004, lists2001.May 27 2024, 9:00 PM