MaxConnTrack Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1060:9100
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jan 15 2024, 1:12 PM

Description

dashboard: https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking
description: Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1060:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack
summary: Max conntrack at 90% on cloudvirt1060:9100

dashboard: https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking
description: Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1060:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack
summary: Max conntrack at 90% on cloudvirt1060:9100
alertname: MaxConnTrack
cluster: wmcs
instance: cloudvirt1060:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

		Status	Subtype	Assigned	Task
		Resolved		taavi	T355061 MaxConnTrack Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1060:9100
		Resolved		taavi	T355067 wmcs-drain-hypervisor is broken

This is a new alert, but seems like an issue regardless.

Mentioned in SAL (#wikimedia-cloud-feed) [2024-01-15T14:27:59Z] <taavi@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot (T355061)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-01-15T14:32:27Z] <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) (T355061)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-01-15T14:32:55Z] <taavi@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot (T355061)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-01-15T14:37:07Z] <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) (T355061)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-01-16T11:02:06Z] <taavi@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot (T355061)

Mentioned in SAL (#wikimedia-cloud-feed) [2024-01-16T11:22:47Z] <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) (T355061)

Cautiously resolving after a reboot.