Cirrus SUP: Unknown error 429 circuit_breaking_exception
Open, Needs TriagePublic5 Estimated Story Points
Actions

Assigned To

Authored By

	• dcausse
	Jul 10 2026, 7:30 AM

Project Tags

Referenced Files

None

Subscribers

Aklapper

bking

• dcausse

Description

The updater in codfw failed a dozen bulk items with:
bulk action failed with status TOO_MANY_REQUESTS: {"index":"commonswiki_file_1764596737","id":"181715533","cause":{"type":"exception","reason":"OpenSearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [indices:data/write/bulk[s]] would be [30604723228/28.5gb], which is larger than the limit of [30601641984/28.5gb], real usage: [30604714848/28.5gb], new bytes reserved: [8380/8.1kb], usages [request=442368/432kb, fielddata=205379902/195.8mb, in_flight_requests=2807510/2.6mb]]","durability":"PERMANENT"},"status":429}.
It happened for several wikis.

This failure should be handled gracefully by the updater.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	cirus: bump to latest image version	Ebernhardson	operations/deployment-charts	master	+1 -1

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Retry individual bulk items	repos/search-platform/cirrus-streaming-updater!207	ebernhardson	work/ebernhardson/429-retry	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		EBernhardson	T431784 Cirrus SUP: Unknown error 429 circuit_breaking_exception
		Open		atsuko	T431827 cirrussearch: alert for tripping circuitbreakers

Event Timeline

• dcausse created this task.Jul 10 2026, 7:30 AM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptJul 10 2026, 7:30 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

bking subscribed.Jul 10 2026, 1:04 PM

atsuko created subtask T431827: cirrussearch: alert for tripping circuitbreakers.Jul 10 2026, 1:15 PM

atsuko mentioned this in T431827: cirrussearch: alert for tripping circuitbreakers.

atsuko changed the status of subtask T431827: cirrussearch: alert for tripping circuitbreakers from Open to In Progress.Jul 10 2026, 3:07 PM

bking mentioned this in T324335: Index the OpenSearch server application logs in the central observability cluster, using ECS format.Jul 10 2026, 4:57 PM

atsuko changed the status of subtask T431827: cirrussearch: alert for tripping circuitbreakers from In Progress to Open.Mon, Jul 13, 9:45 AM

EBernhardson moved this task from needs triage to 2026.06.01 - 2026.07.03 on the Discovery-Search board.Mon, Jul 13, 3:40 PM

EBernhardson edited projects, added: Discovery-Search (2026.06.01 - 2026.07.03); removed: Discovery-Search.

EBernhardson renamed this task from Cirrus update: Unknown error 429 circuit_breaking_exception to Cirrus SUP: Unknown error 429 circuit_breaking_exception.Mon, Jul 13, 3:43 PM

EBernhardson set the point value for this task to 3.Mon, Jul 13, 3:47 PM

EBernhardson moved this task from Incoming to Ready for Dev on the Discovery-Search (2026.06.01 - 2026.07.03) board.

EBernhardson claimed this task.Mon, Jul 13, 8:26 PM

EBernhardson moved this task from Ready for Dev to In Progress on the Discovery-Search (2026.06.01 - 2026.07.03) board.

ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/207

Retry individual bulk items

EBernhardson changed the point value for this task from 3 to 5.Mon, Jul 27, 1:54 PM

ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/207

Retry individual bulk items

Maintenance_bot removed a project: Patch-For-Review.Wed, Jul 29, 2:32 PM

EBernhardson moved this task from In Progress to To be Deployed on the Discovery-Search (2026.06.01 - 2026.07.03) board.Wed, Jul 29, 2:44 PM

Merged and released, but we'll want to wait for monday now to deploy it.