ProbeDown
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Aug 31 2025, 7:20 PM

Description

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
runbook: https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: gerrit1003:443 failed when probed by http_gerrit_tls_ip4 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_gerrit_tls_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443
summary: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4)
address: 208.80.154.135
alertname: ProbeDown
family: ip4
instance: gerrit1003:443
job: probes/custom
module: http_gerrit_tls_ip4
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services-releng
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: gerrit1003:443 failed when probed by http_gerrit_tls_ip6 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_gerrit_tls_ip6)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443
summary: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip6)
address: 2620:0:861:2:208:80:154:135
alertname: ProbeDown
family: ip6
instance: gerrit1003:443
job: probes/custom
module: http_gerrit_tls_ip6
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services-releng
Source

Status	Assigned	Task
Open	None	T407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
Open	None	T407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
Open	None	T392448 Upgrade to Gerrit 3.12
Open	None	T379714 Upgrade to Gerrit 3.11
Open	None	T392465 Switch Gerrit from Java 17 to Java 21
Open	None	T384595 Upgrade Collab hosts to Bookworm
Open	None	T392464 Upgrade Gerrit hosts from Bullseye to Bookworm
Resolved	Jelto	T399306 upgrade gitlab hosts to bookworm
Open	None	T399714 GitLab partman config is unreliable
Resolved	ABran-WMF	T400121 Gitlab-replica switchover (gitlab1004 → gitlab1003)
Resolved	Jelto	T400252 Gitlab switchover (gitlab2002 → gitlab1004)
Resolved	ABran-WMF	T400971 Troubleshoot GitLab nftables throttling after switchover
		Restricted Task
Resolved	ABran-WMF	T403347 ProbeDown
Resolved	Dzahn	T401191 SystemdUnitFailed - wmf_auto_restart_ssh-gitlab.service on gitlab2002
Resolved	Jelto	T401261 SystemdUnitFailed (rsync-data-backup-gitlab2002)

ABran-WMF closed this task as Resolved.Sep 1 2025, 6:42 AM

ABran-WMF claimed this task.

ABran-WMF added a parent task: Restricted Task.