SystemdUnitFailed (planet and gitlab)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Mar 7 2024, 9:49 PM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state

alertname: SystemdUnitFailed
prometheus: ops
severity: critical
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: backup-restore.service on gitlab1003:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: backup-restore.service on gitlab1003:9100
alertname: SystemdUnitFailed
instance: gitlab1003:9100
name: backup-restore.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: backup-restore.service on gitlab1004:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: backup-restore.service on gitlab1004:9100
alertname: SystemdUnitFailed
instance: gitlab1004:9100
name: backup-restore.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: prometheus-apache-exporter.service on planet2003:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: prometheus-apache-exporter.service on planet2003:9100
alertname: SystemdUnitFailed
instance: planet2003:9100
name: prometheus-apache-exporter.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: wmf_auto_restart_prometheus-apache-exporter.service on planet2003:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: wmf_auto_restart_prometheus-apache-exporter.service on planet2003:9100
alertname: SystemdUnitFailed
instance: planet2003:9100
name: wmf_auto_restart_prometheus-apache-exporter.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: prometheus-apache-exporter.service on planet1003:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: prometheus-apache-exporter.service on planet1003:9100
alertname: SystemdUnitFailed
instance: planet1003:9100
name: prometheus-apache-exporter.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: wmf_auto_restart_prometheus-apache-exporter.service on planet1003:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: wmf_auto_restart_prometheus-apache-exporter.service on planet1003:9100
alertname: SystemdUnitFailed
instance: planet1003:9100
name: wmf_auto_restart_prometheus-apache-exporter.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

Details

	Subject	Repo	Branch	Lines +/-
	prometheus/apache_exporter: drop argument parameter	operations/puppet	production	+2 -7

Customize query in gerrit

Related Objects

Mentioned Here: T359556: Add apache exporter to planet hosts

Event Timeline

phaultfinder created this task.Mar 7 2024, 9:49 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 7 2024, 9:49 PM

phaultfinder updated the task description. (Show Details)Mar 8 2024, 2:39 AM

phaultfinder updated the task description. (Show Details)Mar 8 2024, 3:59 AM

GitLab alerts are expected due to version upgrade. But probably we need some kind of silence or workaround to not get 24h of alerts for that.

For planet the exporter fails to start, see T359556#9614359.

GitLab restore errors are resolved by manually running sudo systemctl start backup-restore.service. I think the delay between syncing the backup and restoring is not enough when there are multiple backups for one day (when we've done an upgrade). So the older backup is used which has the wrong version. I'm not sure if we need a followup for that, probably low priority. Just waiting a full 24h would resolve this automatically.

Dzahn merged a task: T359621: SystemdUnitFailed.Mar 8 2024, 3:51 PM

Dzahn merged a task: T359655: SystemdUnitFailed.Mar 8 2024, 7:59 PM

Change 1009775 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] prometheus/apache_exporter: fix argument syntax in bookworm

https://gerrit.wikimedia.org/r/1009775

gerritbot added a project: Patch-For-Review.Mar 8 2024, 8:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-08T20:46:47Z] <mutante> planet1003/2003: apt-get remove prometheus-apache-exporter - T359596

This was due to work in T359556 and a GitLab upgrade, two unrelated work items. I'm closing the task

Change #1009775 merged by Dzahn:

[operations/puppet@production] prometheus/apache_exporter: drop argument parameter