ProbeDown - *_experimental_wikidata_org
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jan 17 2024, 11:18 PM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All

alertname: ProbeDown
family: ip4
job: probes/custom
prometheus: ops
severity: task
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb2003:443 failed when probed by http_query_full_experimental_wikidata_org_collab_ip4 from codfw. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_query_full_experimental_wikidata_org_collab_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb2003:443
summary: Service miscweb2003:443 has failed probes (http_query_full_experimental_wikidata_org_collab_ip4)
address: 10.192.48.65
alertname: ProbeDown
family: ip4
instance: miscweb2003:443
job: probes/custom
module: http_query_full_experimental_wikidata_org_collab_ip4
prometheus: ops
severity: task
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb2003:443 failed when probed by http_query_main_experimental_wikidata_org_collab_ip4 from codfw. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_query_main_experimental_wikidata_org_collab_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb2003:443
summary: Service miscweb2003:443 has failed probes (http_query_main_experimental_wikidata_org_collab_ip4)
address: 10.192.48.65
alertname: ProbeDown
family: ip4
instance: miscweb2003:443
job: probes/custom
module: http_query_main_experimental_wikidata_org_collab_ip4
prometheus: ops
severity: task
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb2003:443 failed when probed by http_query_scholarly_experimental_wikidata_org_collab_ip4 from codfw. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_query_scholarly_experimental_wikidata_org_collab_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb2003:443
summary: Service miscweb2003:443 has failed probes (http_query_scholarly_experimental_wikidata_org_collab_ip4)
address: 10.192.48.65
alertname: ProbeDown
family: ip4
instance: miscweb2003:443
job: probes/custom
module: http_query_scholarly_experimental_wikidata_org_collab_ip4
prometheus: ops
severity: task
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb1003:443 failed when probed by http_query_full_experimental_wikidata_org_collab_ip4 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_query_full_experimental_wikidata_org_collab_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb1003:443
summary: Service miscweb1003:443 has failed probes (http_query_full_experimental_wikidata_org_collab_ip4)
address: 10.64.48.113
alertname: ProbeDown
family: ip4
instance: miscweb1003:443
job: probes/custom
module: http_query_full_experimental_wikidata_org_collab_ip4
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb1003:443 failed when probed by http_query_main_experimental_wikidata_org_collab_ip4 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_query_main_experimental_wikidata_org_collab_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb1003:443
summary: Service miscweb1003:443 has failed probes (http_query_main_experimental_wikidata_org_collab_ip4)
address: 10.64.48.113
alertname: ProbeDown
family: ip4
instance: miscweb1003:443
job: probes/custom
module: http_query_main_experimental_wikidata_org_collab_ip4
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb1003:443 failed when probed by http_query_scholarly_experimental_wikidata_org_collab_ip4 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_query_scholarly_experimental_wikidata_org_collab_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb1003:443
summary: Service miscweb1003:443 has failed probes (http_query_scholarly_experimental_wikidata_org_collab_ip4)
address: 10.64.48.113
alertname: ProbeDown
family: ip4
instance: miscweb1003:443
job: probes/custom
module: http_query_scholarly_experimental_wikidata_org_collab_ip4
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: collaboration-services
Source

Related Objects
Search...

Status	Assigned	Task
Open	None	T335067 Epic: Wikidata Query Service stabilization
Open	None	T337013 [Epic] Splitting the graph in WDQS
Resolved	Gehel	T350464 Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph
Resolved	RKemper	T351650 Expose 3 new dedicated WDQS endpoints
Resolved	RKemper	T354658 Create 3 microsites for wdqs full graph, main graph, & scholarly articles
Resolved	Dzahn	T355273 ProbeDown - *_experimental_wikidata_org

Event Timeline

phaultfinder created this task.Jan 17 2024, 11:18 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 17 2024, 11:18 PM

phaultfinder updated the task description. (Show Details)Jan 17 2024, 11:23 PM

This was caused by https://gerrit.wikimedia.org/r/991412

Dzahn renamed this task from ProbeDown to ProbeDown - *_experimental_wikidata_org.Jan 18 2024, 12:37 AM

These are new experimental sites added as part of T354658.

Jelto merged a task: T355305: ProbeDown - *_experimental_wikidata_org.Jan 18 2024, 12:15 PM

We talked about this and https://gerrit.wikimedia.org/r/c/operations/puppet/+/991680 was merged which disables these sites again for now, until Monday.

So the alert should be gone now.

Then we will have a special meeting about adding the sites to the legacy miscweb TSL cert.

prio low, but keeping open.

let's close it once the sites are added back and have been added to the cert.

we can use this to track the work for that.

should be resolved tomorrow once we added the new sites to the envoy TLS certs. This will happen in the team office hours.

the fix for this will be T355593 which is kind of a duplicate ticket but more specific about the root cause

The missing SANs have now been added to the envoy certs and we could confirm the new sites are working now.

ProbeDown - *_experimental_wikidata_orgClosed, ResolvedPublicActions