ProbeDown (gitlab2002)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Apr 12 2024, 11:06 AM

Description

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: gitlab2002:443 failed when probed by http_gitlab_wikimedia_org_ip6 from codfw. Availability is 50%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_gitlab_wikimedia_org_ip6)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#gitlab2002:443
summary: Service gitlab2002:443 has failed probes (http_gitlab_wikimedia_org_ip6)

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: gitlab2002:443 failed when probed by http_gitlab_wikimedia_org_ip6 from codfw. Availability is 50%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_gitlab_wikimedia_org_ip6)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#gitlab2002:443
summary: Service gitlab2002:443 has failed probes (http_gitlab_wikimedia_org_ip6)
address: 2620:0:860:1:208:80:153:8
alertname: ProbeDown
family: ip6
instance: gitlab2002:443
job: probes/custom
module: http_gitlab_wikimedia_org_ip6
prometheus: ops
severity: task
site: codfw
source: prometheus
team: collaboration-services
Source

this was because of the update in T362298. I'm not 100% sure why this is not silenced, as the cookbook creates a downtime of 180 minutes:

self.alerting_hosts.downtimed(self.admin_reason, duration=timedelta(minutes=180))