ProbeDown
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Mar 10 2025, 10:51 AM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
runbook: https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown

address: 2a02:ec80:a100:fe03::2
alertname: ProbeDown
family: ip6
job: probes/custom
prometheus: ops
severity: critical
source: prometheus
team: wmcs

Firing alerts

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: virt.cloudgw.codfw1dev.wikimediacloud.org:0 failed when probed by icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip6 from codfw. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip6)))))
runbook: https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown
summary: Service virt.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip6)
address: 2a02:ec80:a100:fe03::2
alertname: ProbeDown
family: ip6
instance: virt.cloudgw.codfw1dev.wikimediacloud.org:0
job: probes/custom
module: icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip6
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: wan.cloudgw.codfw1dev.wikimediacloud.org:0 failed when probed by icmp_wan_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip6 from codfw. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:icmp_wan_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip6)))))
runbook: https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown
summary: Service wan.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_wan_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip6)
address: 2a02:ec80:a100:fe03::2
alertname: ProbeDown
family: ip6
instance: wan.cloudgw.codfw1dev.wikimediacloud.org:0
job: probes/custom
module: icmp_wan_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip6
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: virt.cloudgw.codfw1dev.wikimediacloud.org:0 failed when probed by icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_eqiad_ip6 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_eqiad_ip6)))))
runbook: https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown
summary: Service virt.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_eqiad_ip6)
address: 2a02:ec80:a100:fe03::2
alertname: ProbeDown
family: ip6
instance: virt.cloudgw.codfw1dev.wikimediacloud.org:0
job: probes/custom
module: icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_eqiad_ip6
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All
description: wan.cloudgw.codfw1dev.wikimediacloud.org:0 failed when probed by icmp_wan_cloudgw_codfw1dev_wikimediacloud_org_from_eqiad_ip6 from eqiad. Availability is 0%.
logs: https://logstash.wikimedia.org/app/dashboards#/view/f3e709c0-a5f8-11ec-bf8e-43f1807d5bc2?_g=(filters:!((query:(match_phrase:(service.name:icmp_wan_cloudgw_codfw1dev_wikimediacloud_org_from_eqiad_ip6)))))
runbook: https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown
summary: Service wan.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_wan_cloudgw_codfw1dev_wikimediacloud_org_from_eqiad_ip6)
address: 2a02:ec80:a100:fe03::2
alertname: ProbeDown
family: ip6
instance: wan.cloudgw.codfw1dev.wikimediacloud.org:0
job: probes/custom
module: icmp_wan_cloudgw_codfw1dev_wikimediacloud_org_from_eqiad_ip6
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Revert "cloudgw/icmp check/ip6: disabling"	operations/puppet	production	+1 -1
	cloudgw/icmp check/ip6: disabling	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		tappof	T395441 Port all Icinga checks to Prometheus/Alertmanager: preparation
Open		tappof	T321808 Port all Icinga checks to Prometheus/Alertmanager
Resolved		tappof	T370506 Replace check_ripe_atlas with prometheus alert
Resolved	BUG REPORT	tappof	T381561 deployment site for prometheus::blackbox::check::(icmp\|http\|tcp) is not driven by the $site parameter
Resolved		tappof	T381580 cloudgw ICMP checks are affected by changes to prometheus::blackbox::check::icmp
Open		None	T53494 Use Beta cluster as a true canary for code deployments (epic)
Declined		None	T87220 Minimize infrastructure differences between Beta Cluster and production
Open		None	T211677 Support IPv6 in beta
Open		None	T404466 Support IPv6-only VMs
Open		None	T392688 Enable IPv6 on Cloud VPS infrastructure services
Resolved		taavi	T379175 Enable IPv6 for the Cloud VPS web proxy
Open		None	T270694 CloudVPS: introduce tenant networks
Resolved		• aborrero	T364725 Migrate Cloud VPS instances to VXLAN based networks
Open		None	T392509 Enable IPv6 for Toolforge services
Resolved		taavi	T392506 Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service
Resolved		taavi	T211575 Enable IPv6 on toolforge.org
Open		None	T220306 Add IPv6 monitoring
Resolved		• aborrero	T37947 Enable IPv6 on CloudVPS
Resolved		• aborrero	T380174 CloudVPS: IPv6 in eqiad1
Resolved		cmooney	T388379 ProbeDown

Event Timeline

phaultfinder created this task.Mar 10 2025, 10:51 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2025, 10:51 AM

phaultfinder updated the task description. (Show Details)Mar 10 2025, 11:21 AM

irc logs:

11:15:30    tappof │ arturo: dcaro Just a heads-up in case of any unwanted alerts: I'm merging this patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1100819
11:15:59    arturo │ tappof: thanks 🚢🇮🇹
11:15:59     dcaro │ tappof: ack!
11:52:06    arturo │ tappof: indeed we just had the alert firing
11:52:21    arturo │ https://usercontent.irccloud-cdn.com/file/SlS4y6la/image.png
11:52:34    arturo │ and T388379
11:52:34 +stashbot │ T388379: ProbeDown  - https://phabricator.wikimedia.org/T388379
11:52:46    tappof │ arturo: yes, I've seen
11:54:52    arturo │ tappof: are you able to investigate? I'm about to jump into a meeting
11:55:17    tappof │ arturo: yes, I'll check soon
11:55:22    arturo │ thanks
12:02:21       <-- │ dwalden (~dwalden@dwalden.co.uk) has quit (Quit: ZNC 1.8.2+deb2+b1 - https://znc.in)
12:02:37       --> │ dwalden [dwalden] (ZNC - https://znc.in) (~dwalden@81.187.41.249) has joined #wikimedia-cloud
12:07:25       --> │ PhantomTech [PhantomTech] (en:User:PhantomTech) (~PhantomTe@wikipedia/PhantomTech) has joined #wikimedia-cloud
12:34:05        -- │ tgr|away is now known as tgr_
13:06:28    tappof │ arturo: We didn't have the IPv6 check before... https://w.wiki/DNG7 AFAICS, Prometheus in eqiad can reach cloudgw2002-dev on the IPv6 VIP, but the reply gets lost
                   │ somewhere https://snipboard.io/XJ0gn9.jpg
13:06:57    arturo │ topranks: oh, ok!
13:07:40    arturo │ we may want to create a ticket to investigate why that happens, and remove the IPv6 check meanwhile
13:10:05    tappof │ arturo: I think we can use T388379 as the task for this one. I'll link it to the task related to monitoring to keep track of the relationship.
13:10:06 +stashbot │ T388379: ProbeDown  - https://phabricator.wikimedia.org/T388379
13:10:29    tappof │ is it ok for you arturo ?
13:11:26    arturo │ ok!

Change #1126023 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] cloudgw/icmp check/ip6: disabling

https://gerrit.wikimedia.org/r/1126023

gerritbot added a project: Patch-For-Review.Mar 10 2025, 12:15 PM

Change #1126023 merged by Tiziano Fogli:

[operations/puppet@production] cloudgw/icmp check/ip6: disabling

https://gerrit.wikimedia.org/r/1126023

Maintenance_bot removed a project: Patch-For-Review.Mar 10 2025, 12:30 PM

I referenced the wrong task on this patch, but it should allow the pings to work

https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1126035

• aborrero added a parent task: T380174: CloudVPS: IPv6 in eqiad1.Mar 10 2025, 2:10 PM

The alert is not firing anymore, can this be resolved?

In T388379#10623112, @fnegri wrote:

The alert is not firing anymore, can this be resolved?

The alert is not firing because the check was disabled. But the underlying problem is not fixed yet.

We can wait a few days until all patches are merged, in particular https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1126035 and a revert of https://gerrit.wikimedia.org/r/1126023

This should be ok now if you want to try re-enabling the check.

cmooney@prometheus2005:~$ ping -c 2 -6 wan.cloudgw.codfw1dev.wikimediacloud.org.
PING wan.cloudgw.codfw1dev.wikimediacloud.org.(wan.cloudgw.codfw1dev.wikimediacloud.org (2a02:ec80:a100:fe03::2)) 56 data bytes
64 bytes from wan.cloudgw.codfw1dev.wikimediacloud.org (2a02:ec80:a100:fe03::2): icmp_seq=1 ttl=62 time=0.964 ms
64 bytes from wan.cloudgw.codfw1dev.wikimediacloud.org (2a02:ec80:a100:fe03::2): icmp_seq=2 ttl=62 time=0.316 ms

--- wan.cloudgw.codfw1dev.wikimediacloud.org. ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.316/0.640/0.964/0.324 ms

Change #1129770 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Revert "cloudgw/icmp check/ip6: disabling"

https://gerrit.wikimedia.org/r/1129770

gerritbot added a project: Patch-For-Review.Mar 20 2025, 8:40 AM

Change #1129770 merged by Majavah:

[operations/puppet@production] Revert "cloudgw/icmp check/ip6: disabling"

https://gerrit.wikimedia.org/r/1129770

taavi closed this task as Resolved.Mar 20 2025, 9:11 AM

taavi assigned this task to cmooney.

taavi added a project: Cloud-VPS.

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2025, 9:30 AM

ProbeDown Closed, ResolvedPublicActions