Wikimedia sites not reachable through CenturyLink ISP
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	AxelBoldt
	Sep 12 2015, 6:49 PM

Description

I am using the Century Link ISP from Saint Paul, MN. No Wikimedia sites can be reached, other sites work fine.

Reports on http://www.isitdownrightnow.com/wikipedia.org.html indicate that this affects other CenturyLink customers and has been going on for 45 hours.

Two tracert ouputs, taken 30 minutes apart:

$ tracert en.wikipedia.org
Tracing route to en.wikipedia.org [208.80.154.224]over a maximum of 30 hops:
  1     2 ms     1 ms     2 ms  192.168.0.1
  2    44 ms    41 ms    44 ms  207.225.140.237
  3    44 ms    48 ms    56 ms  mpls-agw1.inet.qwest.net [75.168.229.97]
  4    54 ms    54 ms    59 ms  chp-brdr-03.inet.qwest.net [67.14.8.194]
  5    55 ms    51 ms    55 ms  ae5.chi12.ip4.gtt.net [173.205.63.229]
  6    78 ms    79 ms    78 ms  xe-0-3-1.was10.ip4.gtt.net [89.149.128.141]
  7     *        *        *     Request timed out.
  8     *        *      299 ms  text-lb.eqiad.wikimedia.org [208.80.154.224]
Trace complete.

$ tracert en.wikipedia.org
Tracing route to en.wikipedia.org [208.80.154.224]over a maximum of 30 hops:
  1     2 ms     1 ms     1 ms  192.168.0.1
  2    47 ms    41 ms    41 ms  207.225.140.237
  3    48 ms    46 ms    42 ms  mpls-agw1.inet.qwest.net [75.168.229.97]
  4    50 ms    51 ms    57 ms  chp-brdr-03.inet.qwest.net [67.14.8.194]
  5    51 ms    55 ms    50 ms  ae5.chi12.ip4.gtt.net [173.205.63.229]
  6    77 ms    86 ms    78 ms  et-4-3-0.was10.ip4.gtt.net [89.149.129.210]
  7     *      303 ms   291 ms  xe-5-3-1.cr2-eqiad.wikimedia.org [173.241.131.218]
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
 10     *      295 ms     *     text-lb.eqiad.wikimedia.org [208.80.154.224]
 11     *        *        *     Request timed out.
 12     *        *      321 ms  text-lb.eqiad.wikimedia.org [208.80.154.224]
Trace complete.

Event Timeline

AxelBoldt created this task.Sep 12 2015, 6:49 PM

AxelBoldt updated the task description. (Show Details)

Restricted Application added a project: acl*sre-team. · View Herald TranscriptSep 12 2015, 6:49 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Krenair added a project: netops.Sep 12 2015, 8:21 PM

Krenair subscribed.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptSep 12 2015, 8:21 PM

Please provide your IP for a reverse traceroute — you can omit the last number for privacy reasons if you want.

Also, it'd be helpful to know if the site is completely unreachable or just very slow to load. From the traceroute it looks the latter but I'd like to confirm.

I tried ping/traceroute from CenturyLink's Minneapolis PoP using their looking glass. It looks normal so far:

PING 208.80.154.224 (208.80.154.224): 56 data bytes
64 bytes from 208.80.154.224: icmp_seq=0 ttl=60 time=38.781 ms
64 bytes from 208.80.154.224: icmp_seq=1 ttl=60 time=38.781 ms
64 bytes from 208.80.154.224: icmp_seq=2 ttl=60 time=38.730 ms
64 bytes from 208.80.154.224: icmp_seq=3 ttl=60 time=38.780 ms
64 bytes from 208.80.154.224: icmp_seq=4 ttl=60 time=38.791 ms
64 bytes from 208.80.154.224: icmp_seq=5 ttl=60 time=38.735 ms
64 bytes from 208.80.154.224: icmp_seq=6 ttl=60 time=38.739 ms
64 bytes from 208.80.154.224: icmp_seq=7 ttl=60 time=38.758 ms
64 bytes from 208.80.154.224: icmp_seq=8 ttl=60 time=38.769 ms
64 bytes from 208.80.154.224: icmp_seq=9 ttl=60 time=38.757 ms

--- 208.80.154.224 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 38.730/38.762/38.791/0.021 ms

@AxelBoldt could you verify if this is still a problem for you?

I also created a RIPE Atlas measurement from 24 probes that are in CenturyLink's network (AS209).

That's measurement 2407575 and I see nothing abnormal in it so far.

It is now 4:18 PM CDT and I can reach all Mediawiki sites again.

OK. It's impossible to find out what happened without more data unfortunately :(

I'm going to resolve this for now but feel free to reopen (or open a new task) if something like this happens again. Thanks for the report.

faidon closed this task as Resolved.Sep 13 2015, 10:21 AM

faidon claimed this task.

Wikimedia sites not reachable through CenturyLink ISPClosed, ResolvedPublicActions

Description

Event Timeline

Wikimedia sites not reachable through CenturyLink ISP
Closed, ResolvedPublic
Actions