Maniphest T370098

SystemdUnitFailed - lists2001 - sync-list-members
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jul 15 2024, 7:05 PM

Tags

Referenced Files

None

Subscribers

Description

Common information

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state

alertname: SystemdUnitFailed
instance: lists2001:9100
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: sync-list-members-global-renamers.service on lists2001:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: sync-list-members-global-renamers.service on lists2001:9100
alertname: SystemdUnitFailed
instance: lists2001:9100
name: sync-list-members-global-renamers.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: sync-list-members-global-sysops.service on lists2001:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: sync-list-members-global-sysops.service on lists2001:9100
alertname: SystemdUnitFailed
instance: lists2001:9100
name: sync-list-members-global-sysops.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: sync-list-members-stewards-l.service on lists2001:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: sync-list-members-stewards-l.service on lists2001:9100
alertname: SystemdUnitFailed
instance: lists2001:9100
name: sync-list-members-stewards-l.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: sync-list-members-stewards-usergroup.service on lists2001:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: sync-list-members-stewards-usergroup.service on lists2001:9100
alertname: SystemdUnitFailed
instance: lists2001:9100
name: sync-list-members-stewards-usergroup.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

Related Objects

Duplicates Merged Here: T370169: SystemdUnitFailed - lists2001 - sync-list-members
T370140: SystemdUnitFailed - lists2001 - sync-list-members

Event Timeline

phaultfinder created this task.Jul 15 2024, 7:05 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 15 2024, 7:05 PM

Jelto renamed this task from SystemdUnitFailed to SystemdUnitFailed (lists2001).Jul 16 2024, 7:38 AM

LSobanski renamed this task from SystemdUnitFailed (lists2001) to SystemdUnitFailed - lists2001 - sync-list-members.Jul 16 2024, 9:58 AM

LSobanski merged a task: T370140: SystemdUnitFailed - lists2001 - sync-list-members.Jul 16 2024, 1:02 PM

comes from https://gerrit.wikimedia.org/r/c/operations/puppet/+/1053399

works fine on lists1004 but needs to be handled on inactive list server..

need to add code to ensure it's only enabled on active server

Dzahn triaged this task as Medium priority.Jul 16 2024, 2:50 PM

Dzahn merged a task: T370169: SystemdUnitFailed - lists2001 - sync-list-members.Jul 16 2024, 3:53 PM

Dzahn subscribed.

fix: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1054610/2/modules/profile/manifests/lists/automation.pp

Mentioned in SAL (#wikimedia-operations) [2024-07-16T17:00:35Z] <mutante> lists2001 - systemctl reset-failed after gerrit:1054610 to fix T370098

[lists2001:~] $ sudo systemctl list-units --state=failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.