Maniphest T393523

Investigate CirrusSearch "Pool queue is full" errors
Open, MediumPublic
Actions

Assigned To

None

Authored By

	bking
	May 6 2025, 11:01 PM

Project Tags

Referenced Files

None

Subscribers

Description

Per IRC conversation with @Tgr and @EBernhardson , there seems to be an elevated amount of CirrusSearch errors related to the PoolCounter.

Creating this ticket to:

See if we can relate this to our existing dashboard metrics/alerts. For example, the pool counter rejections panel on our dashboard or the Pool Counter Rejection Rate alert for Data Platform SREs

Fix the errors if possible

Improve monitoring/alerting if necessary

Event Timeline

bking created this task.May 6 2025, 11:01 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 6 2025, 11:01 PM

Gehel moved this task from needs triage to 2025.05.02 - 2025.05.23 on the Discovery-Search board.May 12 2025, 2:52 PM

Gehel edited projects, added: Discovery-Search (2025.05.02 - 2025.05.23); removed: Discovery-Search.

Running into pool limitations is by design:

pool counter is related to spikes in traffic, not much we can do about this
thread pools issues are related to hot nodes, not a ton we can do either.

Maybe we can improve the logspam aspect of this.

Gehel removed a project: Discovery-Search (2025.05.02 - 2025.05.23).May 12 2025, 3:41 PM

Gehel triaged this task as Medium priority.May 16 2025, 8:56 AM

Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.

Gehel moved this task from Scratch to Observability on the Data-Platform-SRE board.