Page MenuHomePhabricator

Telkom/8ta (South Africa) users cannot connect to wikimedia sites
Closed, ResolvedPublic

Description

A user on IRC reported that it's not possible to connect to Wikimedia websites since approximately 15 hours ago through this ISP. Apparently, other users of this ISP are also unable to connect. IP range affected is 41.151.0.0/16.

Event Timeline

Peachey88 added a project: netops.
Peachey88 subscribed.

@Glaisher: If the user, or another one experiencing the same issue are still on IRC, could you please ask them to complete a traceroute and attach it to the ticket to assist the ops them?

Glaisher raised the priority of this task from High to Unbreak Now!.Sep 10 2016, 9:15 AM
Glaisher added subscribers: faidon, BBlack.
Glaisher lowered the priority of this task from Unbreak Now! to High.Sep 10 2016, 9:16 AM

Hi everybody, and thanks for all the help!

Here is the ping:
Pinging en.wikipedia.org [91.198.174.192] with 32 bytes of data:

Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 91.198.174.192:

Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

So it resolves the IP fine but nothing gets through

Here are the details of the tracert:

Tracing route to en.wikipedia.org [91.198.174.192]
over a maximum of 30 hops:

1     *        *        *     Request timed out.
2    38 ms    49 ms    39 ms  172.17.32.13 
3    68 ms    69 ms    69 ms  172.17.32.21 
4    77 ms    91 ms    68 ms  8ta-150-177-01.telkomadsl.co.za [41.150.177.1] 
5    56 ms    69 ms    69 ms  196.25.63.233 
6   208 ms   199 ms   199 ms  196.43.9.86 
7   207 ms   210 ms   209 ms  tenge7-2.var01.ldn04.pccwbtn.net [63.218.243.17] 
8   217 ms   219 ms   229 ms  ae2.cr2-esams.wikimedia.org [80.249.209.176] 
9     *        *        *     Request timed out.

10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

If you need any other information, let me know. I am currently connecting to Wikpedia, Wikimedia and Phabricator via an IP hiding service based in the USA, but it will expire in three days, so I need to find a solution before then. My best guess is that our IP range has been blocked due to hacking attempts.

My own IP address is currently 41.144.175.* so I think the IP range might be larger than the one officially published. I'm trying to track down a service which will reveal their full IP range, but so far no luck. However, I did find this:

https://www.robtex.com/?dns=8ta.com

Telkom is announcing their routes directly to us over AMS-IX via the route servers, not a direct peering. However, it seems that they're blackholing all the traffic we're sending them right over from the first hop (on traceroute not even the first hop is appearing).

I'll contact them about it, but in the meantime I've downprefed the direct route and going via transits now.

@Cadar, could you retry and let us know if things work now? Thanks!

I mailed Telkom's NOC. I'll follow up here when I hear back.

Many thanks for the help! I have disabled my IP hider and seem to be able to connect fine now. I've also asked a couple of other people to check that they can connect properly as well. It doesn't surprise me that Telkom have managed to break Wikipedia, it's pretty much typical for them. The Symantec security certificate on their site expired at the end of last month and it turned out they had updated the site with new subdomains and neglected to reroute the traffic from the old pages. That's the joy of using what are laughingly called "service providers" in the Third World for you.

New tracert results:

Tracing route to wikipedia.org [91.198.174.192]
over a maximum of 30 hops:

1     *        *        *     Request timed out.
2    38 ms    29 ms    39 ms  172.17.32.9 
3    62 ms    69 ms    69 ms  172.17.32.17 
4    58 ms    69 ms    69 ms  8ta-150-177-02.telkomadsl.co.za [41.150.177.2] 
5    76 ms    69 ms    90 ms  196.25.63.234 
6    66 ms    69 ms    69 ms  196.25.7.121 
7   197 ms   189 ms   199 ms  lon-ip-hsll-1-gig-0-1-0.telkom-ipnet.co.za [196.43.9.46] 
8   210 ms   239 ms   199 ms  tenge7-2.var01.ldn04.pccwbtn.net [63.218.243.17] 
9     *        *        *     Request timed out.

10 217 ms 239 ms 229 ms text-lb.esams.wikimedia.org [91.198.174.192]

Trace complete.
So we're back in business! Thanks Faidon, you just saved most of Southern Africa from a long dark time of Wikipedia withdrawals 8-)

faidon claimed this task.

Telkom did not respond despite my repeated emails, but I've just removed the explicit route preference and traceroutes seem to indicate that traffic to 41.144.175.0/24 still works.

@Cadar, if connectivity is broken for you now, please reopen this task — and, of course, if you notice any more troubles, please feel free to open a new task. Thanks!

Hi Faidon, thanks for the support on the access question. We seem to be up
and connected to Wikipedia fine on this side. I've noticed similar issues
with some other sites in the last week or two. Perhaps it was just system
upgrades or something. It's typical that they would just ignore requests
for help like that though.

Cheers,
Jon