GerritHAProxyServiceUnavailable
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jun 10 2026, 10:16 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

Dzahn

phaultfinder

Description

Common information

dashboard: grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15
runbook: https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable

alertname: GerritHAProxyServiceUnavailable
job: tcp_proxy
prometheus: ops
proxy: gerrit_ssh
severity: task
source: prometheus
state: DOWN
team: collaboration-services

Firing alerts

dashboard: grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15
description: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN, less than 2 available proxies in codfw.
runbook: https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable
summary: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in codfw
alertname: GerritHAProxyServiceUnavailable
job: tcp_proxy
prometheus: ops
proxy: gerrit_ssh
severity: task
site: codfw
source: prometheus
state: DOWN
team: collaboration-services
Source

dashboard: grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15
description: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN, less than 2 available proxies in eqiad.
runbook: https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable
summary: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in eqiad
alertname: GerritHAProxyServiceUnavailable
job: tcp_proxy
prometheus: ops
proxy: gerrit_ssh
severity: task
site: eqiad
source: prometheus
state: DOWN
team: collaboration-services
Source

Related Objects

Mentioned Here: T425667: Investigate Gerrit root disk usage and logging

Event Timeline

phaultfinder created this task.Jun 10 2026, 10:16 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 10 2026, 10:16 PM

I had to do a service restart for logging changes for T425667

I downtimed the host using the cookbook but that does not stop this from alerting. I am not sure if it's possible to downtime for maintenance.

22:21 <+jinxer-wm> RESOLVED: [4x] GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - 
                   https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - 
                   grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - 
                   https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable
22:21 <+jinxer-wm> RESOLVED: [2x] GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in codfw - 
                   https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - 
                   grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - 
                   https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable

Dzahn merged a task: T428826: GerritHAProxyBackendUnavailable.Jun 10 2026, 11:00 PM

GerritHAProxyServiceUnavailableClosed, ResolvedPublicActions