40% packet loss on ESAMS
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	AlexisJazz
	Mar 21 2025, 12:50 AM

Description

What happens?:

# ping en.wikipedia.org -c10
PING dyna.wikimedia.org (185.15.59.224) 56(84) bytes of data.
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=1 ttl=55 time=16.1 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=2 ttl=55 time=15.5 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=4 ttl=55 time=12.1 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=6 ttl=55 time=12.2 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=9 ttl=55 time=13.4 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=10 ttl=55 time=13.7 ms

--- dyna.wikimedia.org ping statistics ---
10 packets transmitted, 6 received, 40% packet loss, time 9087ms
rtt min/avg/max/mdev = 12.134/13.825/16.059/1.499 ms

Other information (browser name/version, screenshots, etc.):
Not seeing this issue when pinging Google, also now not seeing it when using VPN (which I needed to post this report):

# ping en.wikipedia.org -c10
PING dyna.wikimedia.org (208.80.154.224) 56(84) bytes of data.
64 bytes from text-lb.eqiad.wikimedia.org (208.80.154.224): icmp_seq=1 ttl=55 time=235 ms
64 bytes from text-lb.eqiad.wikimedia.org (208.80.154.224): icmp_seq=2 ttl=55 time=261 ms
64 bytes from text-lb.eqiad.wikimedia.org (208.80.154.224): icmp_seq=3 ttl=55 time=236 ms
64 bytes from text-lb.eqiad.wikimedia.org (208.80.154.224): icmp_seq=4 ttl=55 time=191 ms
64 bytes from text-lb.eqiad.wikimedia.org (208.80.154.224): icmp_seq=5 ttl=55 time=186 ms
64 bytes from text-lb.eqiad.wikimedia.org (208.80.154.224): icmp_seq=6 ttl=55 time=213 ms
64 bytes from text-lb.eqiad.wikimedia.org (208.80.154.224): icmp_seq=7 ttl=55 time=158 ms
64 bytes from text-lb.eqiad.wikimedia.org (208.80.154.224): icmp_seq=8 ttl=55 time=245 ms
64 bytes from text-lb.eqiad.wikimedia.org (208.80.154.224): icmp_seq=9 ttl=55 time=248 ms
64 bytes from text-lb.eqiad.wikimedia.org (208.80.154.224): icmp_seq=10 ttl=55 time=186 ms

--- dyna.wikimedia.org ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9010ms
rtt min/avg/max/mdev = 157.814/215.854/260.978/32.505 ms

Event Timeline

AlexisJazz created this task.Mar 21 2025, 12:50 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 21 2025, 12:50 AM

AlexisJazz renamed this task from Seeing dropped packets on esams to 40% packetloss on esams.Mar 21 2025, 12:52 AM

AlexisJazz renamed this task from 40% packetloss on esams to 40% packet loss on ESAMS.Mar 21 2025, 12:55 AM

AlexisJazz added projects: ops-esams, SRE.Mar 21 2025, 12:58 AM

Restricted Application added a project: DC-Ops. · View Herald TranscriptMar 21 2025, 12:58 AM

If I specifically ping ESAMS through my VPN, again packet loss:

# ping text-lb.esams.wikimedia.org -c10
PING text-lb.esams.wikimedia.org (185.15.59.224) 56(84) bytes of data.
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=1 ttl=54 time=328 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=2 ttl=54 time=335 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=4 ttl=54 time=231 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=5 ttl=54 time=199 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=6 ttl=54 time=267 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=7 ttl=54 time=216 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=8 ttl=54 time=260 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=10 ttl=54 time=252 ms

--- text-lb.esams.wikimedia.org ping statistics ---
10 packets transmitted, 8 received, 20% packet loss, time 9033ms
rtt min/avg/max/mdev = 199.306/261.045/335.393/45.939 ms

I'm also having some pretty severe packet loss to esams.

ping phabricator.wikimedia.org -c10
PING phabricator.wikimedia.org (185.15.59.224) 56(84) bytes of data.
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=1 ttl=55 time=42.5 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=6 ttl=55 time=18.7 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=7 ttl=55 time=130 ms
64 bytes from text-lb.esams.wikimedia.org (185.15.59.224): icmp_seq=9 ttl=55 time=94.3 ms

--- phabricator.wikimedia.org ping statistics ---
10 packets transmitted, 4 received, 60% packet loss, time 9175ms
rtt min/avg/max/mdev = 18.677/71.276/129.645/43.383 ms

BCornwall changed the task status from Open to In Progress.Mar 21 2025, 1:27 AM

BCornwall triaged this task as Unbreak Now! priority.

Thank you for the report. We've looked into the issue and now the network is behaving properly again.

40% packet loss on ESAMSClosed, ResolvedPublicBUG REPORTActions

Description

Event Timeline

40% packet loss on ESAMS
Closed, ResolvedPublicBUG REPORT
Actions