Maniphest T431827

cirrussearch: alert for tripping circuitbreakers
Open, Needs TriagePublic
Actions

Assigned To

Authored By

	atsuko
	Jul 10 2026, 1:15 PM

Project Tags

Referenced Files

None

Subscribers

Description

Consider if the alerting on tripping memory circuitbreaker is giving useful signal and debuggable. Seems like we had this kind of alert in the past, T355795.

We had circuitbreaker-related issues recently, investigating how to better alert

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Revert "cirrussearch: alert on tripped circuit breakers"	Atsuko	operations/alerts	master	+0 -71
	cirrussearch: alert on tripped circuit breakers	Atsuko	operations/alerts	master	+71 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		EBernhardson	T431784 Cirrus SUP: Unknown error 429 circuit_breaking_exception
		In Progress		BTullis	T324335 Index the OpenSearch server application logs in the central observability cluster, using ECS format
		Open		atsuko	T431827 cirrussearch: alert for tripping circuitbreakers

Event Timeline

atsuko created this task.Jul 10 2026, 1:15 PM

atsuko renamed this task from Setup the alert for triggering circuitbreakers to cirrussearch: alert for tripping circuitbreakers.Jul 10 2026, 1:17 PM

Change #1309685 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/alerts@master] cirrussearch: alert on tripped circuit breakers

https://gerrit.wikimedia.org/r/1309685

gerritbot added a project: Patch-For-Review.Jul 10 2026, 3:06 PM

atsuko changed the task status from Open to In Progress.Jul 10 2026, 3:07 PM

atsuko moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.

atsuko added a project: Data-Platform-SRE (2026-07-03 - 2026-07-31).

Change #1309685 merged by jenkins-bot:

[operations/alerts@master] cirrussearch: alert on tripped circuit breakers

https://gerrit.wikimedia.org/r/1309685

Maintenance_bot removed a project: Patch-For-Review.Jul 10 2026, 4:32 PM

atsuko updated the task description. (Show Details)Jul 10 2026, 5:10 PM

Alert is now acknowledged for a weekend and Monday while I'm drafting the runbook.

Data in prometheus are corresponding to the GC events
It doesn't correlate with Caused by: org.opensearch.core.common.breaker.CircuitBreakingException: [parent] Data too large, ..., which is larger than the limit of [30601641984/28.5gb]

I think this alert might be moderately useless.

atsuko moved this task from In Progress to Backlog - operations on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Mon, Jul 13, 9:15 AM

Change #1309691 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/alerts@master] Revert "cirrussearch: alert on tripped circuit breakers"

https://gerrit.wikimedia.org/r/1309691

gerritbot added a project: Patch-For-Review.Mon, Jul 13, 9:18 AM

Reverted the change, blocked on T324335.

Change #1309691 merged by jenkins-bot:

[operations/alerts@master] Revert "cirrussearch: alert on tripped circuit breakers"

https://gerrit.wikimedia.org/r/1309691

atsuko added a parent task: T324335: Index the OpenSearch server application logs in the central observability cluster, using ECS format.Mon, Jul 13, 9:49 AM

Maintenance_bot removed a project: Patch-For-Review.Mon, Jul 13, 10:31 AM

atsuko updated the task description. (Show Details)Tue, Jul 14, 9:14 AM

atsuko moved this task from Backlog - operations to Blocked/Waiting on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Tue, Jul 14, 1:47 PM

EBernhardson moved this task from needs triage to 2026.06.01 - 2026.07.03 on the Discovery-Search board.Mon, Jul 20, 3:55 PM

EBernhardson edited projects, added: Discovery-Search (2026.06.01 - 2026.07.03); removed: Discovery-Search.

EBernhardson moved this task from Incoming to Blocked / Waiting on the Discovery-Search (2026.06.01 - 2026.07.03) board.Mon, Jul 27, 3:29 PM

pfischer edited projects, added: Discovery-Search (2026.08.03 - 2026.08.31); removed: Discovery-Search (2026.06.01 - 2026.07.03).Mon, Aug 3, 2:41 PM

pfischer moved this task from Incoming to Blocked / Waiting on the Discovery-Search (2026.08.03 - 2026.08.31) board.