Alert in need of triage: Number of requests triggering circuit breakers due to excessive memory usage (instance graphite1005)
Open, HighPublic
Actions

Assigned To

None

Authored By

	LSobanski
	Feb 15 2024, 10:32 AM

Description

The alert Number of requests triggering circuit breakers due to excessive memory usage has started firing 1 month ago.

Labels

alertname=Number of requests triggering circuit breakers due to excessive memory usage
instance=graphite1005
severity=unknown
source=icinga
team=sre

Annotations

Name	Content
dashboard	https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1&var-cluster=eqiad&var-smoothing=1&viewPanel=9
runbook	https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring
summary	UNKNOWN: No valid datapoints found

Links

Triage metadata. Do not delete.

fingerprint=30e964ac5b433a63

Related Objects

Mentioned Here: T355795: Fix "requests triggering circuit breakers" Elastic alert

Event Timeline

LSobanski created this task.Feb 15 2024, 10:32 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 15 2024, 10:32 AM

Looking at T355795: Fix "requests triggering circuit breakers" Elastic alert, this seems like something for the data platform team. Please send it back if not, and we can disable the alert.

Gehel triaged this task as High priority.Feb 29 2024, 9:29 AM

Gehel moved this task from Incoming to 2024.02.12 - 2024.03.03 on the Data-Platform-SRE board.

Gehel edited projects, added Data-Platform-SRE (2024.02.12 - 2024.03.03); removed Data-Platform-SRE.

Gehel edited projects, added Data-Platform-SRE (2024.03.04 - 2024.03.24); removed Data-Platform-SRE (2024.02.12 - 2024.03.03).Mar 4 2024, 1:33 PM

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.03.04 - 2024.03.24) board.Mar 22 2024, 8:46 AM

Gehel edited projects, added Data-Platform-SRE (2024.03.25 - 2024.04.14); removed Data-Platform-SRE (2024.03.04 - 2024.03.24).Mar 22 2024, 8:48 AM

@lmata Let's go ahead and disable this alert. We'll make a new one once the relevant metrics are available in Prometheus. Thanks and sorry for the noise!

bking edited projects, added SRE Observability; removed Data-Platform-SRE (2024.03.25 - 2024.04.14).Apr 2 2024, 3:41 PM

@bking ack! thank you for the confirmation!

@andrea.denisse could you disable this alert please?

lmata moved this task from Inbox to Up next on the SRE Observability (FY2023/2024-Q4) board.Apr 2 2024, 6:58 PM

Alert in need of triage: Number of requests triggering circuit breakers due to excessive memory usage (instance graphite1005)Open, HighPublicActions