ProbeDown - Search Platform still getting these tickets, we need this to stop
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Mon, Dec 2, 8:57 AM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
runbook: https://wikitech.wikimedia.org/wiki/Runbook#wdqs1015:443

alertname: ProbeDown
instance: wdqs1015:443
job: probes/custom
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: search-platform

Firing alerts

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: wdqs1015:443 failed when probed by http_query_wikidata_org_ldf_ip4 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_query_wikidata_org_ldf_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#wdqs1015:443
summary: Service wdqs1015:443 has failed probes (http_query_wikidata_org_ldf_ip4)
address: 10.64.132.7
alertname: ProbeDown
family: ip4
instance: wdqs1015:443
job: probes/custom
module: http_query_wikidata_org_ldf_ip4
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: search-platform
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: wdqs1015:443 failed when probed by http_query_wikidata_org_ldf_ip6 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_query_wikidata_org_ldf_ip6)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#wdqs1015:443
summary: Service wdqs1015:443 has failed probes (http_query_wikidata_org_ldf_ip6)
address: 2620:0:861:10b:10:64:132:7
alertname: ProbeDown
family: ip6
instance: wdqs1015:443
job: probes/custom
module: http_query_wikidata_org_ldf_ip6
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: search-platform
Source

Related Objects

Mentioned Here: T379182: ProbeDown - wdqs1015/migrate LDF alerts to Data Platform SRE

Event Timeline

phaultfinder created this task.Mon, Dec 2, 8:57 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Dec 2, 8:57 AM

Gehel moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.Mon, Dec 2, 4:24 PM

Gehel moved this task from In Progress to DPE-SRE on the Discovery-Search (Current work) board.

Gehel edited projects, added Data-Platform-SRE (2024.11.30 - 2024.12.20); removed Discovery-Search (Current work).

Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2024.11.30 - 2024.12.20) board.Tue, Dec 3, 4:27 PM

bking renamed this task from ProbeDown to ProbeDown - Search Platform still getting these tickets, we need this to stop.Tue, Dec 3, 4:43 PM

bking changed the task status from Open to In Progress.

bking triaged this task as Low priority.

Search platform shouldn't be getting these alerts anyone-Data Platform SRE should be the sole responder. I thought I fixed this in T379182 , but it appears I need to take another look.

I checked the alert files on the Prometheus hosts using cumin:

sudo cumin A:prometheus 'grep search-platform /srv/prometheus/ops/rules/alerts_query_wikidata_org_ldf.yml'

Based on that output, it seems there are no remaining LDF alerts that route to the Search Platform team. That being said, it's always possible that the prometheus services need a restart or reload before the configuration is actually applied. For now, I'm going to assume that everything is fixed and close this ticket. We can always revisit if the alerts come back.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2024.11.30 - 2024.12.20) board.Fri, Dec 20, 1:35 PM

ProbeDown - Search Platform still getting these tickets, we need this to stopClosed, ResolvedPublicActions