Alert in need of triage: PybalBackendDown (instance elastic2090:0)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	LSobanski
	Thu, May 9, 10:31 AM

Description

The alert PybalBackendDown has started firing 1 month ago.

Labels

alertname=PybalBackendDown
instance=elastic2090:0
prometheus=ops
service=search-psi-https_9643
severity=warning
site=codfw
source=prometheus
team=sre

Annotations

Name	Content
dashboard	TODO
description	Pybal has been failing health checks for elastic2090:0 for a long time.
runbook	TODO
summary	Pybal backend elastic2090:0 is down (search-psi-https_9643)

Links

Triage metadata. Do not delete.

fingerprint=e52a94a8354dc4d3

Details

	Subject	Repo	Branch	Lines +/-
	elasticsearch: add elastic2090 to correct pybal pool	operations/puppet	production	+1 -1

Customize query in gerrit

Event Timeline

LSobanski created this task.Thu, May 9, 10:31 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, May 9, 10:31 AM

LSobanski added a project: Data-Platform-SRE.Thu, May 9, 10:56 AM

Gehel triaged this task as High priority.Fri, May 10, 8:21 AM

Gehel edited projects, added Data-Platform-SRE (2024.05.06 - 2024.05.26); removed Data-Platform-SRE.

bking moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.05.06 - 2024.05.26) board.Mon, May 13, 9:58 PM

When I brought this host online a few weeks back, I accidentally added it to the psi pool. I've since fixed this , but my fix seems to be incomplete. Will finish this out when I can, but I did want to share the reason.

bking claimed this task.Fri, May 17, 1:21 PM

Change #1032784 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elasticsearch: add elastic2090 to correct pybal pool

https://gerrit.wikimedia.org/r/1032784

gerritbot added a project: Patch-For-Review.Fri, May 17, 1:42 PM

Change #1032784 merged by Bking:

[operations/puppet@production] elasticsearch: add elastic2090 to correct pybal pool

https://gerrit.wikimedia.org/r/1032784

Maintenance_bot removed a project: Patch-For-Review.Fri, May 17, 2:30 PM

The above prometheus query no longer returns any results, so I believe this is fixed. Closing...

bking closed this task as Resolved.Tue, May 21, 2:23 PM

Alert in need of triage: PybalBackendDown (instance elastic2090:0)Closed, ResolvedPublicActions