Maniphest T400816

SystemdUnitFailed (prometheus-nft-throttling-denylist)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jul 30 2025, 2:13 PM

Project Tags

Referenced Files

None

Subscribers

Description

Common information

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state

alertname: SystemdUnitFailed
name: prometheus-nft-throttling-denylist.service
prometheus: ops
severity: critical
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: prometheus-nft-throttling-denylist.service on gitlab2002:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: prometheus-nft-throttling-denylist.service on gitlab2002:9100
alertname: SystemdUnitFailed
instance: gitlab2002:9100
name: prometheus-nft-throttling-denylist.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: prometheus-nft-throttling-denylist.service on gitlab1003:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: prometheus-nft-throttling-denylist.service on gitlab1003:9100
alertname: SystemdUnitFailed
instance: gitlab1003:9100
name: prometheus-nft-throttling-denylist.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: prometheus-nft-throttling-denylist.service on gitlab1004:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: prometheus-nft-throttling-denylist.service on gitlab1004:9100
alertname: SystemdUnitFailed
instance: gitlab1004:9100
name: prometheus-nft-throttling-denylist.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T384595 Upgrade Collab hosts to Bookworm
		Resolved		Jelto	T399306 upgrade gitlab hosts to bookworm
		Resolved		Jelto	T400252 Gitlab switchover (gitlab2002 → gitlab1004)
		Resolved		Jelto	T400816 SystemdUnitFailed (prometheus-nft-throttling-denylist)

Event Timeline

phaultfinder created this task.Jul 30 2025, 2:13 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 30 2025, 2:13 PM

phaultfinder updated the task description. (Show Details)Jul 30 2025, 2:18 PM

phaultfinder updated the task description. (Show Details)Jul 30 2025, 2:23 PM

this happened because nftables rate-limiting was disabled in T400252 but the exporter was still running. This was fixed by also disabling the exporter.

I'll resolve the task.