create blackbox::http monitoring for RT
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dzahn
	Jan 26 2023, 12:21 AM

Description

follow-up task from T327772

create basic blackbox::http check that automatically creates tickets for RT

https://rt.wikimedia.org

follow existing examples for phabricator/gitlab/vrts linked in ticket above

Details

Subject	Repo	Branch	Lines +/-
requesttracker: limit http monitoring to IPv4, for now	operations/puppet	production	+1 -0
Update status_matches to be array, not int	operations/puppet	production	+1 -1
Add blackbox http check for rt.wikimedia.org	operations/puppet	production	+8 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Dzahn	T327772 Automatically created ProbeDown tasks for Collab services
Resolved	Dzahn	T329587 http::blackbox monitoring for all https services (let all serviceops-collab alertmanager alerts create tickets)
Resolved	eoghan	T327978 create blackbox::http monitoring for RT

Event Timeline

Dzahn created this task.Jan 26 2023, 12:21 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 26 2023, 12:21 AM

LSobanski triaged this task as Medium priority.Jan 30 2023, 4:33 PM

LSobanski moved this task from Incoming to Backlog on the collaboration-services board.

Dzahn added a parent task: T329587: http::blackbox monitoring for all https services (let all serviceops-collab alertmanager alerts create tickets).Feb 16 2023, 7:47 PM

eoghan claimed this task.Mar 13 2023, 3:49 PM

eoghan moved this task from Backlog to Work in Progress on the collaboration-services board.

Change 897917 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Add blackbox http check for rt.wikimedia.org

https://gerrit.wikimedia.org/r/897917

gerritbot added a project: Patch-For-Review.Mar 13 2023, 3:54 PM

Change 897917 merged by EoghanGaffney:

[operations/puppet@production] Add blackbox http check for rt.wikimedia.org

https://gerrit.wikimedia.org/r/897917

Maintenance_bot removed a project: Patch-For-Review.Mar 14 2023, 11:30 AM

eoghan closed this task as Resolved.Mar 14 2023, 8:54 PM

Thank you!:)

Wondering, can you actually find this as "probe success" here? This is an example for the releases* check:

https://thanos.wikimedia.org/graph?g0.deduplicate=0&g0.expr=probe_success%7Binstance%3D~%22.*releases.*%22%7D&g0.max_source_resolution=0s&g0.partial_response=0&g0.range_input=1h&g0.stacked=0&g0.store_matches=%5B%5D&g0.tab=1

Change 898889 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Update status_matches to be array, not int

https://gerrit.wikimedia.org/r/898889

gerritbot added a project: Patch-For-Review.Mar 14 2023, 11:33 PM

Change 898889 merged by EoghanGaffney:

[operations/puppet@production] Update status_matches to be array, not int

https://gerrit.wikimedia.org/r/898889

Maintenance_bot removed a project: Patch-For-Review.Mar 15 2023, 12:30 AM

eoghan reopened this task as Open.Mar 15 2023, 12:40 AM

The probe is showing success now, as seen here

Thank you!:) confirmed, I see it in thanos. (Just wanted to share that option as well.)

for some reason it shows up here now https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=team%3Dserviceops-collab

https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:http_rt_wikimedia_org_ip6)))))&_a=h@cdc7fc8

looked ok in thanos though before

we are refusing the connection via IPv6 (at least coming from prometheus1005)

[prometheus1005:~] $ curl https://[2620:0:861:101:10:64:0:54]
curl: (7) Failed to connect to 2620:0:861:101:10:64:0:54 port 443: Connection refused

[prometheus1005:~] $ curl 10.64.0.54
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>

Change 899807 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] requesttracker: limit http monitoring to IPv4, for now

https://gerrit.wikimedia.org/r/899807

gerritbot added a project: Patch-For-Review.Mar 15 2023, 10:36 PM

Change 899807 merged by Dzahn:

[operations/puppet@production] requesttracker: limit http monitoring to IPv4, for now

https://gerrit.wikimedia.org/r/899807

change above cleared the alert from alerts.wikimedia.org

Dzahn closed this task as Resolved.Mar 15 2023, 10:47 PM

Maintenance_bot removed a project: Patch-For-Review.Mar 15 2023, 11:10 PM