ProbeDown - Make these alerts less noisy.
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jun 1 2024, 12:43 PM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: wdqs1015:443 failed when probed by http_query_wikidata_org_ldf_ip6 from eqiad. Availability is 50%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_query_wikidata_org_ldf_ip6)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#wdqs1015:443
summary: Service wdqs1015:443 has failed probes (http_query_wikidata_org_ldf_ip6)

address: 2620:0:861:10b:10:64:132:7
alertname: ProbeDown
family: ip6
instance: wdqs1015:443
job: probes/custom
module: http_query_wikidata_org_ldf_ip6
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: search-platform

Firing alerts

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: wdqs1015:443 failed when probed by http_query_wikidata_org_ldf_ip6 from eqiad. Availability is 50%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_query_wikidata_org_ldf_ip6)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#wdqs1015:443
summary: Service wdqs1015:443 has failed probes (http_query_wikidata_org_ldf_ip6)
address: 2620:0:861:10b:10:64:132:7
alertname: ProbeDown
family: ip6
instance: wdqs1015:443
job: probes/custom
module: http_query_wikidata_org_ldf_ip6
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: search-platform
Source

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	wdqs: detune blackbox checks	operations/puppet	production	+4 -2

Customize query in gerrit

Related Objects

Mentioned Here: T369227: Explore the usage of "virtual team" (custom receiver) for alerting

Event Timeline

phaultfinder created this task.Jun 1 2024, 12:43 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 1 2024, 12:43 PM

• OKJ04 added a subtask: T366433: CentralAuth tests broken unless you run them inside Quibble.Jun 3 2024, 1:39 AM

JJMC89 removed a subtask: T366433: CentralAuth tests broken unless you run them inside Quibble.Jun 3 2024, 1:44 AM

Gehel edited projects, added Data-Platform-SRE; removed Discovery-Search (Current work).Jun 17 2024, 3:31 PM

Gehel triaged this task as Medium priority.Jun 18 2024, 8:45 AM

Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.

Gehel edited projects, added Data-Platform-SRE (2024.06.17 - 2024.07.07); removed Data-Platform-SRE.Jun 18 2024, 8:48 AM

Can we make sure that the hostname is added to the title of the task?

@Gehel I think so. I've found some examples (such T362360) . I'll check w/collaboration services and see how they work their magic.

@Dzahn helpfully pointed out this comment, so we know it's possible. Asking in observability IRC and will update this ticket when I have more details.

Per IRC conversation with @fgiunchedi , this is feature is not quite ready. It is on the radar for next quarter, so we'll keep an eye out. In the meantime, I'm going to use this ticket to detune these alerts a bit, as they are a bit noisy.

Change #1051369 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: detune blackbox checks

https://gerrit.wikimedia.org/r/1051369

gerritbot added a project: Patch-For-Review.Jul 2 2024, 1:48 PM

bking renamed this task from ProbeDown - Add hostname automatically. to ProbeDown - Make these alerts less noisy..Jul 2 2024, 5:32 PM

bking changed the task status from Open to In Progress.

Change #1051369 merged by Ryan Kemper: