Page MenuHomePhabricator

INMARSAT geolocates to the UK, leading to requests going to esams
Closed, DeclinedPublic

Description

So on a Qatar Airways flight between DOH and AKL, I noticed it was going back to esams. Not sure if this makes sense in all cases

IP 161.30.203.16

Sams-MacBook-Pro:~ reedy$ ping text-lb.esams.wikimedia.org
PING text-lb.esams.wikimedia.org (91.198.174.192): 56 data bytes
64 bytes from 91.198.174.192: icmp_seq=0 ttl=43 time=904.348 ms
64 bytes from 91.198.174.192: icmp_seq=1 ttl=43 time=966.516 ms
64 bytes from 91.198.174.192: icmp_seq=2 ttl=43 time=850.605 ms
64 bytes from 91.198.174.192: icmp_seq=3 ttl=43 time=910.318 ms
^C
--- text-lb.esams.wikimedia.org ping statistics ---
5 packets transmitted, 4 packets received, 20.0% packet loss
round-trip min/avg/max/stddev = 850.605/907.947/966.516/41.040 ms
Sams-MacBook-Pro:~ reedy$ ping text-lb.eqiad.wikimedia.org
PING text-lb.eqiad.wikimedia.org (208.80.154.224): 56 data bytes
64 bytes from 208.80.154.224: icmp_seq=0 ttl=45 time=768.968 ms
64 bytes from 208.80.154.224: icmp_seq=1 ttl=45 time=768.083 ms
64 bytes from 208.80.154.224: icmp_seq=2 ttl=45 time=772.186 ms
64 bytes from 208.80.154.224: icmp_seq=3 ttl=45 time=771.782 ms
64 bytes from 208.80.154.224: icmp_seq=4 ttl=45 time=760.882 ms
64 bytes from 208.80.154.224: icmp_seq=5 ttl=45 time=815.495 ms
64 bytes from 208.80.154.224: icmp_seq=6 ttl=45 time=883.064 ms
64 bytes from 208.80.154.224: icmp_seq=7 ttl=45 time=765.375 ms
^C
--- text-lb.eqiad.wikimedia.org ping statistics ---
8 packets transmitted, 8 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 760.882/788.229/883.064/39.230 ms
Sams-MacBook-Pro:~ reedy$ ping text-lb.codfw.wikimedia.org
PING text-lb.codfw.wikimedia.org (208.80.153.224): 56 data bytes
64 bytes from 208.80.153.224: icmp_seq=0 ttl=46 time=848.402 ms
64 bytes from 208.80.153.224: icmp_seq=1 ttl=46 time=734.402 ms
64 bytes from 208.80.153.224: icmp_seq=2 ttl=46 time=742.595 ms
64 bytes from 208.80.153.224: icmp_seq=3 ttl=46 time=735.485 ms
64 bytes from 208.80.153.224: icmp_seq=4 ttl=46 time=822.874 ms
^C
--- text-lb.codfw.wikimedia.org ping statistics ---
6 packets transmitted, 5 packets received, 16.7% packet loss
round-trip min/avg/max/stddev = 734.402/776.752/848.402/48.835 ms
Sams-MacBook-Pro:~ reedy$ ping text-lb.ulsfo.wikimedia.org
PING text-lb.ulsfo.wikimedia.org (198.35.26.96): 56 data bytes
64 bytes from 198.35.26.96: icmp_seq=0 ttl=45 time=722.297 ms
64 bytes from 198.35.26.96: icmp_seq=1 ttl=45 time=724.097 ms
64 bytes from 198.35.26.96: icmp_seq=2 ttl=45 time=757.644 ms
64 bytes from 198.35.26.96: icmp_seq=3 ttl=45 time=731.146 ms
64 bytes from 198.35.26.96: icmp_seq=4 ttl=45 time=729.734 ms
64 bytes from 198.35.26.96: icmp_seq=5 ttl=45 time=728.418 ms
64 bytes from 198.35.26.96: icmp_seq=6 ttl=45 time=716.117 ms
Request timeout for icmp_seq 7
64 bytes from 198.35.26.96: icmp_seq=7 ttl=45 time=1174.906 ms
64 bytes from 198.35.26.96: icmp_seq=8 ttl=45 time=720.446 ms
64 bytes from 198.35.26.96: icmp_seq=9 ttl=45 time=719.559 ms
64 bytes from 198.35.26.96: icmp_seq=10 ttl=45 time=732.219 ms
^C64 bytes from 198.35.26.96: icmp_seq=11 ttl=45 time=711.023 ms
^C
--- text-lb.ulsfo.wikimedia.org ping statistics ---
13 packets transmitted, 12 packets received, 7.7% packet loss
round-trip min/avg/max/stddev = 711.023/763.967/1174.906/124.404 ms
Sams-MacBook-Pro:~ reedy$ ping text-lb.eqsin.wikimedia.org
PING text-lb.eqsin.wikimedia.org (103.102.166.224): 56 data bytes
64 bytes from 103.102.166.224: icmp_seq=0 ttl=47 time=729.871 ms
64 bytes from 103.102.166.224: icmp_seq=1 ttl=47 time=734.393 ms
64 bytes from 103.102.166.224: icmp_seq=2 ttl=47 time=764.762 ms
64 bytes from 103.102.166.224: icmp_seq=3 ttl=47 time=788.261 ms
64 bytes from 103.102.166.224: icmp_seq=4 ttl=47 time=809.938 ms
^C
--- text-lb.eqsin.wikimedia.org ping statistics ---
6 packets transmitted, 5 packets received, 16.7% packet loss
round-trip min/avg/max/stddev = 729.871/765.445/809.938/30.758 ms
Sams-MacBook-Pro:~ reedy$

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

When looking at the latest MaxMind data, it locates this network as being in New Zealand, which we map to ulsfo as first choice, and esams as the last-resort choice. But the destination would've been set by geodns logic, so probably what really mattered was the location of the DNS cache in use. For future debugging, try a DNS lookup on reflect.wikimedia.org, which will show us what DNS cache exit IP our servers see, e.g.:

$ dig +short reflect.wikimedia.org
192.0.2.1

Same IP going back, 161.30.203.16

Reedys-MacBook-Pro:~ reedy$ dig +short reflect.wikimedia.org
161.30.203.0

But it does seem to be going to ulsfo now, I guess after having been in NZ, this is potentially more expected if there's some caching going on from my laptop. I wonder if on the outbound, as I'd been in europe, that explains the usage of esams as being cached on my laptop

On the flight for a few more hours...

jijiki triaged this task as Low priority.EditedDec 3 2018, 1:27 PM
jijiki subscribed.

@Reedy Should we mark this as "Resolved"? What do you think? :)