ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Mar 29 2023, 11:40 PM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All

alertname: ProbeDown
family: ip4
job: probes/custom
prometheus: ops
severity: task
source: prometheus
team: serviceops-collab

Firing alerts

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb2002:443 failed when probed by http_commons_query_wikimedia_org_ip4 from codfw. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_commons_query_wikimedia_org_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb2002:443
summary: Service miscweb2002:443 has failed probes (http_commons_query_wikimedia_org_ip4)
address: 10.192.16.211
alertname: ProbeDown
family: ip4
instance: miscweb2002:443
job: probes/custom
module: http_commons_query_wikimedia_org_ip4
prometheus: ops
severity: task
site: codfw
source: prometheus
team: serviceops-collab
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb2002:443 failed when probed by http_transparency_wikimedia_org_ip4 from codfw. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_transparency_wikimedia_org_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb2002:443
summary: Service miscweb2002:443 has failed probes (http_transparency_wikimedia_org_ip4)
address: 10.192.16.211
alertname: ProbeDown
family: ip4
instance: miscweb2002:443
job: probes/custom
module: http_transparency_wikimedia_org_ip4
prometheus: ops
severity: task
site: codfw
source: prometheus
team: serviceops-collab
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb2003:443 failed when probed by http_commons_query_wikimedia_org_ip4 from codfw. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_commons_query_wikimedia_org_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb2003:443
summary: Service miscweb2003:443 has failed probes (http_commons_query_wikimedia_org_ip4)
address: 10.192.48.65
alertname: ProbeDown
family: ip4
instance: miscweb2003:443
job: probes/custom
module: http_commons_query_wikimedia_org_ip4
prometheus: ops
severity: task
site: codfw
source: prometheus
team: serviceops-collab
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb2003:443 failed when probed by http_transparency_wikimedia_org_ip4 from codfw. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_transparency_wikimedia_org_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb2003:443
summary: Service miscweb2003:443 has failed probes (http_transparency_wikimedia_org_ip4)
address: 10.192.48.65
alertname: ProbeDown
family: ip4
instance: miscweb2003:443
job: probes/custom
module: http_transparency_wikimedia_org_ip4
prometheus: ops
severity: task
site: codfw
source: prometheus
team: serviceops-collab
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb1002:443 failed when probed by http_commons_query_wikimedia_org_ip4 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_commons_query_wikimedia_org_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb1002:443
summary: Service miscweb1002:443 has failed probes (http_commons_query_wikimedia_org_ip4)
address: 10.64.32.187
alertname: ProbeDown
family: ip4
instance: miscweb1002:443
job: probes/custom
module: http_commons_query_wikimedia_org_ip4
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: serviceops-collab
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb1002:443 failed when probed by http_transparency_wikimedia_org_ip4 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_transparency_wikimedia_org_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb1002:443
summary: Service miscweb1002:443 has failed probes (http_transparency_wikimedia_org_ip4)
address: 10.64.32.187
alertname: ProbeDown
family: ip4
instance: miscweb1002:443
job: probes/custom
module: http_transparency_wikimedia_org_ip4
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: serviceops-collab
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb1003:443 failed when probed by http_commons_query_wikimedia_org_ip4 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_commons_query_wikimedia_org_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb1003:443
summary: Service miscweb1003:443 has failed probes (http_commons_query_wikimedia_org_ip4)
address: 10.64.48.113
alertname: ProbeDown
family: ip4
instance: miscweb1003:443
job: probes/custom
module: http_commons_query_wikimedia_org_ip4
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: serviceops-collab
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: miscweb1003:443 failed when probed by http_transparency_wikimedia_org_ip4 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_transparency_wikimedia_org_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#miscweb1003:443
summary: Service miscweb1003:443 has failed probes (http_transparency_wikimedia_org_ip4)
address: 10.64.48.113
alertname: ProbeDown
family: ip4
instance: miscweb1003:443
job: probes/custom
module: http_transparency_wikimedia_org_ip4
prometheus: ops
severity: task
site: eqiad
source: prometheus
team: serviceops-collab
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: contint2002:1443 failed when probed by http_integration_wikimedia_org_ip4 from codfw. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_integration_wikimedia_org_ip4)))))
runbook: https://wikitech.wikimedia.org/wiki/Runbook#contint2002:1443
summary: Service contint2002:1443 has failed probes (http_integration_wikimedia_org_ip4)
address: 208.80.153.39
alertname: ProbeDown
family: ip4
instance: contint2002:1443
job: probes/custom
module: http_integration_wikimedia_org_ip4
prometheus: ops
severity: task
site: codfw
source: prometheus
team: serviceops-collab
Source

Details

Subject	Repo	Branch	Lines +/-
microsites: do not expect 301 for commons-query.wikimedia.org	operations/puppet	production	+6 -8
microsites: commons-query.wm.org only works on port 80/http	operations/puppet	production	+1 -0
microsites: for transparency.wikimedia.org expect HTTP 302, not 301	operations/puppet	production	+1 -1
microsites: do not use TLS when monitoring commons-query.wikimedia.org	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Dzahn	T333510 clean up commons-query on miscweb setup
		Resolved		Dzahn	T333507 ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb)

Event Timeline

phaultfinder created this task.Mar 29 2023, 11:40 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 29 2023, 11:40 PM

phaultfinder updated the task description. (Show Details)Mar 29 2023, 11:45 PM

filed T333510 for the root cause of this (for https://commons-query.wikimedia.org)

Change 904377 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] microsites: do not use TLS when monitoring commons-query.wikimedia.org

https://gerrit.wikimedia.org/r/904377

gerritbot added a project: Patch-For-Review.Mar 30 2023, 12:24 AM

Change 904377 merged by Dzahn:

[operations/puppet@production] microsites: do not use TLS when monitoring commons-query.wikimedia.org

https://gerrit.wikimedia.org/r/904377

Change 904378 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] microsites: for transparency.wikimedia.org except HTTP 302, not 301

https://gerrit.wikimedia.org/r/904378

Change 904378 merged by Dzahn:

[operations/puppet@production] microsites: for transparency.wikimedia.org expect HTTP 302, not 301

https://gerrit.wikimedia.org/r/904378

https://gerrit.wikimedia.org/r/c/operations/puppet/+/904374

Change 904382 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] microsites: commons-query.wm.org only works on port 80/http

https://gerrit.wikimedia.org/r/904382

Change 904382 merged by Dzahn:

[operations/puppet@production] microsites: commons-query.wm.org only works on port 80/http

https://gerrit.wikimedia.org/r/904382

Dzahn renamed this task from ProbeDown to ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb).Mar 30 2023, 12:56 AM

Change 904384 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] microsites: do not expect 301 for commons-query.wikimedia.org

https://gerrit.wikimedia.org/r/904384

Dzahn mentioned this in T333515: ProbeDown - commons_query_wikimedia_org.Mar 30 2023, 1:12 AM

Change 904384 merged by Dzahn:

[operations/puppet@production] microsites: do not expect 301 for commons-query.wikimedia.org

https://gerrit.wikimedia.org/r/904384

also see T333510

finally everthing ok in https://thanos.wikimedia.org/graph?g0.deduplicate=1&g0.expr=probe_success%7Binstance%3D~%22.*miscweb.*%22%7D&g0.max_source_resolution=0s&g0.partial_response=0&g0.range_input=1h&g0.stacked=0&g0.store_matches=%5B%5D&g0.tab=1

Dzahn closed this task as Resolved.Mar 30 2023, 1:23 AM

Dzahn claimed this task.

Dzahn added a parent task: T333510: clean up commons-query on miscweb setup.