Page MenuHomePhabricator

Connection problem (Moscow ISP, 4G) with Beeline / Sovintel
Closed, ResolvedPublic

Description

Hello! During the last month, users of a large mobile ISP (4G) have reported about problem connection to the Wikipedia site (and through apps) . A very lot of reports about it (OTRS, social networks, ruwiki forum, private conversations).

The Supports checked everything on side of ISP and recommends to all users contact the "Supports of Wikimedia". They claim that the problem is on our side.

Further, according to the information of one of the letters to OTRS:
ISP: Beeline (Moscow, 4G only):
IP: 83.220.238.125
ping:

Pinging ru.wikipedia.org [91.198.174.192] with 32 bytes of data:
Reply from 91.198.174.192: bytes=32 time=68ms TTL=54
Reply from 91.198.174.192: bytes=32 time=76ms TTL=54
Reply from 91.198.174.192: bytes=32 time=100ms TTL=54
Reply from 91.198.174.192: bytes=32 time=87ms TTL=54
Packets: Sent =  4, Received = 4, Lost = 0 (0% lost)

tracert:

Tracing route to ru.wikipedia.org [91.198.174.192] over a maximum is 30 hops:
  1     2 ms     1 ms     2 ms  192.168.43.1
  2     *        *        *     Request timed out.
  3    42 ms    20 ms    43 ms  10.10.31.21
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6    37 ms    27 ms    39 ms  10.10.31.121
  7    31 ms    30 ms    32 ms  10.10.31.198
  8    34 ms    23 ms    27 ms  62.105.150.252
  9    73 ms    67 ms    67 ms  mx01.Amsterdam.gldn.net [79.104.225.148]
10    70 ms    70 ms    69 ms  ae2.cr2-esams.wikimedia.org [80.249.209.176]
11    74 ms    64 ms    62 ms  text-lb.esams.wikimedia.org [91.198.174.192]

curl:

-v https://ru.wikipedia.org/wiki/Main_Page > C:/curl-wikipedia.html
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 91.198.174.192...
* TCP_NODELAY set
0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0* connect to 91.198.174.192 port 443 failed: Timed out
* Failed to connect to ru.wikipedia.org port 443: Timed out
* Closing connection 0
curl: (7) Failed to connect to ru.wikipedia.org port 443: Timed out

[C:/curl-wikipedia.html is empty.]

Please check everything's. Maybe IP range is blocked at the server level or something else.
Sorry for my English.

Relevant threads at ru.wiki:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

See [wikitech] for the full list"

Do you mean "Additional troubleshooting" sectional and curl to http? Is this a completely full list or maybe you need something else? The problem is massive, but this is the first user who agreed to run cURL. We should not lose him :)

What is the issue in your own words: see above.
Is the issue intermittent or constant: constant
When did the issue start: 1 month ago
What device are you using: mobile and PC (via mobile as access point)
What network are you connected to? Beeline (only 4G network), Russia, Moscow and Moscow region
What is your public IP: see above
cURL: see above
traceroute: see above
ping: see above

Adding that a, from our esams DC, traceroute to this IP seems to stop before beelive.ru

$ traceroute 83.220.238.125
traceroute to 83.220.238.125 (83.220.238.125), 30 hops max, 60 byte packets
 1  ae1-100.cr2-esams.wikimedia.org (91.198.174.3)  0.607 ms  0.579 ms  0.559 ms
 2  mx01.Amsterdam.gldn.net (80.249.209.153)  0.862 ms  3.235 ms  3.224 ms
 3  pe05.KK12.Moscow.gldn.net (79.104.225.13)  41.235 ms  41.473 ms pe05.KK12.Moscow.gldn.net (79.104.225.15)  39.832 ms
 4  62.105.150.251 (62.105.150.251)  41.938 ms  39.834 ms  40.083 ms
 5  * * *

And the last IP in the hop is from a company

inetnum:        62.105.150.248 - 62.105.150.255
netname:        RU-SOVINTEL-MSK-Poliplastik-Group-NET
descr:          119034 Russia Moscow, ZC-5724343
descr:          Ochakovskoe sh, 18, KL-1943598
descr:          POLYPLASTIC GROUP Ltd., http://www.polyplastic.ru
country:        RU
admin-c:        PFoA1-RIPE
tech-c:         PFoA1-RIPE
status:         ASSIGNED PA
mnt-by:         SOVINTEL-MNT
created:        2012-07-20T08:49:57Z
last-modified:  2012-07-20T08:49:57Z
source:         RIPE # Filtered

That handles plastic pipes per http://www.polyplastic.ru/ ?

The AS of beelive.ru is AS16345 and is advertised fine to us per our BGP routing tables. The AS of that company is AS8350 and is NOT part of the AS path we have for 83.220.238.125. This is starting to look like misconfiguration at Sovintel (the upstream of polyplastic judging by whois, and probably one of beelines.ru upstreams as well)

Aklapper renamed this task from Connection problem (Moscow ISP, 4G) to Connection problem (Moscow ISP, 4G) with Beeline / Sovintel.Jan 23 2019, 12:40 PM

I think the "poliplastic" track a red herring, even though those few IPs are assigned to them in whois, it's used by AS3216, AS16345's upstream, most likely as router IPs.
The fact that the traceroute from us to 83.220.238.125 stops is probably due to the NAT gateway (or something on the way) blocking ICMP.
The fact that pings and traceroutes from them to us works, indicates that routing is most likely fine.

My guess so far would be a middle box (tcp optimization or similar) not working as expected.
We know that https to text-lb.esams times out, now we can try to test other sites/protocols.

Please try:

  • http with the same endpoint: curl -v http://en.wikipedia.org/wiki/Main_Page
  • https on a different site: curl -v https://text-lb.ulsfo.wikimedia.org --insecure
  • http on a different site: curl -v http://text-lb.ulsfo.wikimedia.org
  • https on the same site but different endpoint: curl -v https://upload-lb.esams.wikimedia.org --insecure
  • http on the same site but different endpoint: curl -v http://upload-lb.esams.wikimedia.org

Once we have those information, we can think of a next step.

Note for anyone who wants to help, please make sure you're on the same network that the one having issues, for that browse https://bgp.he.net/ and make sure it says Your ISP is AS16345

Ciao @ayounsi!
It is possible that CommRel can help you finding someone to diagnose this tomorrow if @Iluvatar doesn't happen to find others to test in the meantime.

(edited to remove confusing information.)

Please try:

  • http with the same endpoint: curl -v http://en.wikipedia.org/wiki/Main_Page
  • https on a different site: curl -v https://text-lb.ulsfo.wikimedia.org --insecure
  • http on a different site: curl -v http://text-lb.ulsfo.wikimedia.org
  • https on the same site but different endpoint: curl -v https://upload-lb.esams.wikimedia.org --insecure
  • http on the same site but different endpoint: curl -v http://upload-lb.esams.wikimedia.org

See https://disk.yandex.by/d/Scrhfdy0BBYdAQ

(Putting this in different order so that it's potentially more understandable by other people without a deep tech knowledge, who may be willing to contribute information:)

Step 0:

Note for anyone who wants to help, please make sure you're on the same network that the one having issues, for that browse https://bgp.he.net/ and make sure it says Your ISP is AS16345

Step 1: Head to https://wikitech.wikimedia.org/wiki/Reporting_a_connectivity_issue to learn how to file an accurate report if you haven't done before.

Step 2:

Please try:

  • http with the same endpoint: curl -v http://en.wikipedia.org/wiki/Main_Page
  • https on a different site: curl -v https://text-lb.ulsfo.wikimedia.org --insecure
  • http on a different site: curl -v http://text-lb.ulsfo.wikimedia.org
  • https on the same site but different endpoint: curl -v https://upload-lb.esams.wikimedia.org --insecure
  • http on the same site but different endpoint: curl -v http://upload-lb.esams.wikimedia.org

Once we have those information, we can think of a next step.

Thank you, it's very curious.

Could you try curl http://www.google.com and curl https://www.google.com --insecure as control? (to be sure other sites work as expected).
Then telnet 198.35.26.96 80 and telnet 198.35.26.96 443 and telnet 91.198.174.113 22

Problem solved. The user received today a new answer to one of many his letters to Support: "we reloaded GPRS settings". Connection is restored. 1 month later...
Thanks to all.

ayounsi claimed this task.

Good to see a happy ending here. Thanks to everyone who helped.