SystemdUnitFailed - lists2001
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	May 23 2024, 12:24 PM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state

alertname: SystemdUnitFailed
instance: lists2001:9100
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: mailman3.service on lists2001:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: mailman3.service on lists2001:9100
alertname: SystemdUnitFailed
instance: lists2001:9100
name: mailman3.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: stewards_subscriber_data_sync.service on lists2001:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: stewards_subscriber_data_sync.service on lists2001:9100
alertname: SystemdUnitFailed
instance: lists2001:9100
name: stewards_subscriber_data_sync.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

Related Objects
Search...

Status	Assigned	Task
Open	None	T291916 Tracking task for Bullseye migrations in production
Resolved	Dzahn	T327068 Bullseye upgrade for remaining Collab hosts
Resolved	• eoghan	T331706 Migrate Mailman/lists to Bullseye/Bookworm
Resolved	• eoghan	T283615 Make mailman3 work in the standby host (lists2001.wikimedia.org)
Resolved	• eoghan	T365698 SystemdUnitFailed - lists2001

Event Timeline

phaultfinder created this task.May 23 2024, 12:24 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 23 2024, 12:24 PM

phaultfinder updated the task description. (Show Details)May 23 2024, 1:09 PM

Jelto renamed this task from SystemdUnitFailed to SystemdUnitFailed (lists2001).May 23 2024, 1:29 PM

Dzahn added a parent task: T283615: Make mailman3 work in the standby host (lists2001.wikimedia.org).May 23 2024, 6:14 PM

LSobanski renamed this task from SystemdUnitFailed (lists2001) to SystemdUnitFailed - lists2001.May 27 2024, 5:57 AM

This happened as part of the deployment of mailman3 on the new hosts. It fails because there are firewall rules stopping the new hosts from communicating with the mysql db. This fails when the package is installed, and tries to be started by the package installer, but since we want the package to be stopped on the host, we can tolerate this failure temporarily.

• eoghan mentioned this in T365781: SystemdUnitFailed - gitlab1003, gitlab1004, lists2001.May 27 2024, 8:48 PM

• eoghan mentioned this in T365999: SystemdUnitFailed - lists1004.May 27 2024, 8:53 PM

SystemdUnitFailed - lists2001Closed, ResolvedPublicActions