We decided to file https://github.com/citizenlab/test-lists/pull/730 so that we can get some test data.

(Thanks Chris for suggesting the use of this specific image to help incorporate some other debugging data as well.)

@ssingh did we get results from those test data?

@ssingh: Any news here?

@ayounsi, @Aklapper: I forgot to update this ticket, apologies! Last I checked, all reports were in the green and there wasn't any debugging information. But since we have had no user updates and it's been a while since I looked, I will check again and update this ticket accordingly. Thanks!

There is evidence in our NEL data that suggests this problem still exists.

This particular issue ought to manifest as reports of type tcp.reset. tcp.reset events can occur for a variety of reasons, so there's plenty of background noise here.

Looking at all tcp.reset reports received from India, AS134674 doesn't hugely stand out as a source of these reports -- it's in line with large ISPs:

However, when you look at the breakdowns by server IP address and for domain name, and how they differ between these ISPs, things get interesting. Here's an example for Jio (but Airtel and Vodafone India look quite similar to this).

For whatever reason, there's a higher ambient rate of resets for IPv6 connections, and said rate is fairly even between the text and upload addresses.

Here's the Jio view of resets by domain name:

Now here's the same two plots for Tata Sky Broadband:

In Tata's case, the data is wildly skewed towards upload-lb, regardless of IPv4 vs IPv6, and, the per-domain data is incredibly skewed towards upload.wm.o.

Additionally, the total level of reports from Tata Sky is itself suggestive of an issue -- per our webrequest data, while the rate of tcp.reset error reports from Tata users is the same order of magnitude as what we receive from Jio or Airtel, the total overall request volume from Tata Sky users is only 1-2% of the volume from either Jio or Airtel (!).

This issue has been reported on Znuny on Ticket#2021091710000804, so it's safe to assume the problem is still present.

Recent email exchange on this as they contacted us again:

From: Cathal Mooney
Sent: Thursday, September 16, 2021, 9:42 AM
To: TataSkyBB NOC
Cc: noc; info-en; TataSkyBB NOC; L3 Servicedesk; DL-Technology
Subject: Wikipedia images are not getting open_KOL region

Hi,

Thank you for your mail. I am sorry to hear your users are having
problems using Wikipedia.

There was a lengthy email thread between your team and ourselves
starting back last January, which I've attached to this mail.

At the time it was clear, from a supplied pcap you sent, that your
user's TLS requests to "upload.wikimedia.org" were being blocked by
some device / middlebox in the path, and not reaching us. This was
clear as the PCAP showed the client receiving responses from "our" IP
with ~10ms RTT. But when we traceroute to your IP space from the
serving Wikipedia node (in Singapore) the actual RTT is over 50ms. In
each case as soon as the TLS Client Hello was sent by your client, a
fake TCP RST packet was sent to it purporting to come from our IP,
approx 10ms after the Client Hello was sent. There is no way this is
really a response from our server, as it could not have got to us and
back in such a short time. So the packet is clearly being injected by
some device intercepting the flow. I've attached a pcap with just a
single example of this taken from the file you sent us back then.

Did you make any progress in identifying the middlebox / device
responsible for this behaviour? Is the situation now different than
at that point? If you believe the situation is different please let
us know what was done to resolve the issue the last time, and also
provide the following:

• Public IP of the client making the request in the PCAP (if they are

behind NAT).

• Dig / DNS lookup to "dyna.wikimedia.org"
• Dig / DNS lookup to "reflect.wikimedia.org"
• Traceroute to "dyna.wikimedia.org"
• PCAP of all that and of an attempt to use upload.wikimedia.org from a browser

Please be advised if the situation is the same as last time it is
completely out of our control and we will be unable to assist. All we
could say is to try to find and remove, or reconfigure, the middlebox
responsible for blocking.

Kind regards,

Cathal Mooney
SRE Infrastructure Foundations
Wikimedia Foundation.
https://wikimediafoundation.org/

On Thu, 16 Sept 2021 at 06:18, TataSkyBB NOC wrote:

Dear Team,

We are unable to open images on wikipedia from Kolkata region. Snap is attached for reference.

Kindly help to whitelist below IP’s (Ipv4 and Ipv6) on https://upload.wikimedia.org as per geographic location Kolkata to load the image properly.

IPv4 -

103.59.72.0/24

103.59.73.0/24

45.119.30.0/24

45.127.44.0/24

45.127.45.0/24

103.195.202.0/24

IPV6-

2402:e280:2400::/40

2402:e280:3d00::/40

AS No. 134674

Regards,

Removing Traffic as I don't think this looks actionable for our team (but might still be for netops if the conversations above are still ongoing!).

In T275234#7361977, @Perryprog wrote:

This issue has been reported on Znuny on Ticket#2021091710000804, so it's safe to assume the problem is still present.

And this user states that they're now seeing images again! :)

Per NEL data it looks like that this issue was fixed on Sept 17th!

These are the reported TCP resets for Tata Sky Broadband Private Limited (AS134674). As before, they were overwhelmingly for upload.wikimedia.org.