Page MenuHomePhabricator

Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons
Closed, ResolvedPublic

Assigned To
Authored By
Joe
Wed, Feb 3, 11:21 AM
Tokens
"Burninate" token, awarded by Harej."The World Burns" token, awarded by DannyS712."Barnstar" token, awarded by Asartea."Barnstar" token, awarded by valerio.bozzolan."Barnstar" token, awarded by jijiki."Barnstar" token, awarded by mmodell."Cookie" token, awarded by Ladsgroup."Y So Serious" token, awarded by Prtksxna."Meh!" token, awarded by KartikMistry."The World Burns" token, awarded by Amire80."Cup of Joe" token, awarded by Elitre.

Description

Please avoid adding drive-by comments such as "hello from Hacker News" to this task as they are not helpful. Thank you.

We've noticed today that we get about 90M requests per day from various ISPs in India, all with the same characteristics:

URL: https://upload.wikimedia.org/wikipedia/commons/thumb/1/16/AsterNovi-belgii-flower-1mb.jpg/1280px-AsterNovi-belgii-flower-1mb.jpg
Referer: "-"
User-Agent: "-"

These are very strange, as they come from wildly different IPs, follow a daily traffic pattern, so we are hypothesising there is some mobile app predominantly used in india that hotlinks the above image for e.g. a splash screen.

We need to investigate this further as this kind of requests consitutes about 20% of all requests we get in EQSIN for media.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I don't have much knowledge about India's internet infrastructure but from experience of Iran and blocking apps/websites. They show you a page from a reserved IP (so it's not accessible to the outside) saying "Sorry, this page is not accessible in Iran". It might be part of that page and we can't see it because of that. Specially given that the raise coincides with the block of TikTok in India. What happens if you try to access TikTok (or any other blocked app/website) in India? I totally understand Iran't "internet" is different from what most countries have so I might be talking rubbish here.

Xxpor added a subscriber: Xxpor.Tue, Feb 9, 12:44 AM

Going to the TikTok website from India results in the regular TikTok page loading, with a banner from TikTok saying that the service is unavailable in India. Not a dedicated block page.

Screenshot

(I have access to proxy servers in India).

Dzahn added a subscriber: Dzahn.EditedTue, Feb 9, 1:13 AM

https://newshimalaya.com/2021/02/09/%E2%9A%93-t273741-investigate-unusual-media-traffic-pattern-for-asternovi-belgii-flower-1mb-jpg-on-commons/

^ wut? I tried to search for links to this image and found... this Phabricator ticket content on a Nepali news site?

varenc added a subscriber: varenc.Tue, Feb 9, 1:35 AM

Going to the TikTok website from India results in the regular TikTok page loading, with a banner from TikTok saying that the service is unavailable in India. Not a dedicated block page.

The lack of a User-Agent and all other distinguishing headers means this can't be coming from a web browser.

I would look up the IPs and see if some of them are from IP blocks associated with cellular providers. Since in general only mobile phones are on those IPs, that'll provide be some strong evidence that this is from a mobile app. My guess is an online connectivity check? I assume the detailed IP/request logs aren't public or I'd go investigate this myself. 🙃

Izno added a subscriber: Izno.Tue, Feb 9, 3:09 AM
tomglynch added a subscriber: tomglynch.EditedTue, Feb 9, 3:16 AM

Hi all, I've been doing a bit of research into possible apps that could be causing this and found two potential culprits that I am currently investigating.

The first is Mitron TV, (news article here), an Indian TikTok alternative which was made available again on the app store June 6th.

The second is Say Namaste, (news article here), an Indian Zoom alternative which was launched on the app stores June 9th.

Both fall into the timeline of huge increases, have millions of users and may be using '1280px-AsterNovi-belgii-flower-1mb.jpg' to check the users internet connection - especially for Say Namaste to ensure video connectivity. I've reached out to some developers at both companies and will report back. Let me know your thoughts.

EDIT: I have also noticed the dates match the reopening after lockdown for the whole of India: "This first phase of reopening was termed as "Unlock 1.0"[13] and permitted shopping malls, religious places, hotels and restaurants to reopen from 8 June." from Wikipedia

Tom

ssingh added a comment.Tue, Feb 9, 3:29 AM

Thank you everyone for the comments and suggestions. I just wanted to share that we have identified the app and will update this task tomorrow. (And yes, it is a mobile app.)

Chlod added a subscriber: Chlod.Tue, Feb 9, 3:51 AM
mfkp69 added a subscriber: mfkp69.Tue, Feb 9, 4:02 AM
Phuzion added a subscriber: Phuzion.Tue, Feb 9, 4:10 AM
mmodell added a subscriber: mmodell.Tue, Feb 9, 4:33 AM

^ wut? I tried to search for links to this image and found... this Phabricator ticket content on a Nepali news site?

Looks like it's just a big rss aggregator?

This comment was removed by mmodell.
rootkea added a subscriber: rootkea.Tue, Feb 9, 4:54 AM

Hello From us Hacker News readers ;)

Ltrlg added a subscriber: Ltrlg.Tue, Feb 9, 7:17 AM
Zardula added a subscriber: Zardula.Tue, Feb 9, 7:27 AM
This comment was removed by Zardula.
Majavah updated the task description. (Show Details)Tue, Feb 9, 7:30 AM
miyuru added a subscriber: miyuru.Tue, Feb 9, 7:45 AM
R4356th added a subscriber: R4356th.Tue, Feb 9, 8:27 AM
Shizhao added a subscriber: Shizhao.Tue, Feb 9, 9:00 AM

Rename to a new filename?

MBH added a subscriber: MBH.Tue, Feb 9, 1:34 PM
wkandek added a subscriber: wkandek.Tue, Feb 9, 1:52 PM
lmata added a subscriber: lmata.Tue, Feb 9, 2:34 PM
rabbbit added a subscriber: rabbbit.Tue, Feb 9, 3:08 PM

Rename to a new filename?

The Commons community generally avoids moving files, as it can break attribution and cause issues for (reasonable) external reusers. "Turn it off and see who screams" is a valid method when less disruptive methods fail, but it appears to be unnecessary in this case. Others have previously suggested serving a different file for users with the matching user agent header, which wouldn't break all external links to the file. While that solution would require more work than simply moving the file, it may also have been more effective (depending on how it was used).

Mvolz updated the task description. (Show Details)Tue, Feb 9, 4:01 PM

Change 663004 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] upload-frontend: ban a specific url with no referer nor UA

https://gerrit.wikimedia.org/r/663004

Chlod removed a subscriber: Chlod.Tue, Feb 9, 4:29 PM
ssingh added a comment.Tue, Feb 9, 5:17 PM

Update:

Thank you for the interest in this task! Like we shared yesterday, we have identified that the traffic is coming from a popular mobile app in India. We have initiated contact with the app developers, and are waiting to hear back from them. In the meantime, given the volume of requests, we have decided to ban those specific requests until the issue is resolved. While we will refrain from naming the app at this time, we can share that it is not on the list of apps mentioned in this task. Nevertheless, we thank you for your comments and suggestions on how to debug this!

Since there has been some interest in how we narrowed it down to this particular app:

  1. The header attributes suggested that this was a mobile app. We then queried Hive, we determined that the connection attributes related to these headers (User-Agent and Referer) were mostly from IPv6 addresses, further confirming the theory that this was a popular mobile app.
  2. We then tried to isolate connections from geographical regions and ISPs in India but it was clear that there was no pattern there, as users were spread across the country.
  3. A few things were clear given the volume of the requests: it was a popular app with traffic throughout the day (and even late at night) with a peak on December 31 2020, suggesting that it may be a chat or social media app.
  4. We noticed that the image/app gained popularity somewhere around the time India blocked Chinese internet services and websites, thus affecting popular apps in India like TikTok. (This was pointed out by a user.)
  5. Based on the information above, we gathered a list of popular chat and social media applications in the country, especially apps that gained popularity after the above censorship event.
  6. We first started by downloading and running these apps to see if we could identify the image in their splash screens or within the apps. We also asked the community on the ground and there are many unnamed people who helped us with this -- thank you!
  7. This unfortunately didn't work as none of the apps we tested had the image anywhere -- neither in the splash screen nor in the apps themselves. The community in India was equally surprised given the popularity of this image/app and the fact that they had not seen it in their daily usage.
  8. It was then speculated that the app fetches the image but does not show it. (This was based on this comment.)
  9. To recap, we were aware of the following at this stage:
    • it is a popular chat/social media mobile app used in India
    • it sets the User-Agent and Referer to '-'
    • it fetches the image from Wikimedia Commons but does not display it
  10. To narrow down the app, we decided to observe connections to the image from clients (phones) to our servers. We did this by opening the popular apps one-by-one and noting down the time. After doing this for all the apps, we then ran this query in Hive: SELECT * FROM wmf.webrequest WHERE year=2021 AND month=2 AND day=9 AND parse_media_file_url(uri_path).base_name='/wikipedia/commons/1/16/AsterNovi-belgii-flower-1mb.jpg' AND webrequest_source='upload' AND uri_host = 'upload.wikimedia.org' AND user_agent='-' AND ip=<IP>;
  11. We then found the specific app that was making the request by matching the time when it was opened and the time image was requested from our servers, restricting the results to the User-Agent '-' and from the IP we tested.
  12. By this time, we had isolated the app and were convinced that this is the one that is fetching the image on startup. We could not find the image anywhere in the app, confirming our theory that it fetches the image but does not display it.
  13. To further confirm this finding and to ensure that we had the correct app, we decided to log DNS queries from a phone by setting up a local resolver to capture DNS traffic. After pointing the phone towards it and launching the app, we noticed that it was indeed the one looking up upload.wikimedia.org on startup.
fdans added a comment.Tue, Feb 9, 5:23 PM

@ssingh said it yesterday on chat but this is such stellar data detective work. Congrats on finding the culprit!!

Is the effect that the block will have in the app known?

rootkea removed a subscriber: rootkea.Tue, Feb 9, 5:25 PM
jijiki awarded a token.Tue, Feb 9, 5:40 PM
calbon added a subscriber: calbon.Tue, Feb 9, 5:49 PM

Just to second what @fdans said, the data detective work was great and this was such a fun ticket to watch.

Joe added a comment.Tue, Feb 9, 6:02 PM

Is the effect that the block will have in the app known?

No, hence we tried to reach out to them, although it seems that there is no good way to get in touch with them through email (I sent an email to all publicly available channels, only to get back an autoresponder that assumes I'm an user of the app and asking for my phone number). I eventually resorted to DM their CEO on twitter.

I think this is more than enough courtesy on our part. Anyways, the block will clearly link to this task as the reason for the block, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/663004/4/modules/varnish/templates/upload-frontend.inc.vcl.erb#380, so that whoever is responsible for this can figure out how to reach us.

The traffic spikes are closely matching indian holidays. 2 Oct, 5 sept, 14 Nov, 31 Dec, 12-14 Jan etc.

amy_rc added a subscriber: amy_rc.Wed, Feb 10, 10:44 AM
Joe added a comment.EditedWed, Feb 10, 8:43 PM

Is the effect that the block will have in the app known?

No, hence we tried to reach out to them, although it seems that there is no good way to get in touch with them through email (I sent an email to all publicly available channels, only to get back an autoresponder that assumes I'm an user of the app and asking for my phone number). I eventually resorted to DM their CEO on twitter.

I think this is more than enough courtesy on our part. Anyways, the block will clearly link to this task as the reason for the block, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/663004/4/modules/varnish/templates/upload-frontend.inc.vcl.erb#380, so that whoever is responsible for this can figure out how to reach us.

Small - and happy! - status update: we managed to get in touch with the app developers, who confirmed our suspicions were correct, and promised a quick turnaround in posting an updated version of the application. We will thus hold back the banning of the url for now, awaiting for confirmation of the desired effect to reduce the potential harmful impact on the application users.

Given how much "sample code" we found around the internet using that url, it might still be a good idea to merge the patch later just to prevent this from happening again.

Arrbee added a subscriber: Arrbee.Thu, Feb 11, 10:19 AM
Joe added a comment.Fri, Feb 12, 7:27 AM

According to data in turnilo, the flow of requests has gone down significantly after the app authors fixed the issue on their side:

I'm inclined to resolve the task once we have a longer-range confirmation that the traffic is indeed gone for good.

Our traffic also decreased significantly in Eqsin.

Sdkb added a subscriber: Sdkb.Fri, Feb 12, 7:36 AM

Just a quick heads up, apparently Vice found this phab ticket worthy of an article: https://www.vice.com/en/article/qjpmyx/why-is-this-flower-on-wikipedia-suddenly-getting-90-million-hits-per-day

Mvolz added a comment.Fri, Feb 12, 4:16 PM

Is the effect that the block will have in the app known?

No, hence we tried to reach out to them, although it seems that there is no good way to get in touch with them through email (I sent an email to all publicly available channels, only to get back an autoresponder that assumes I'm an user of the app and asking for my phone number). I eventually resorted to DM their CEO on twitter.

I think this is more than enough courtesy on our part. Anyways, the block will clearly link to this task as the reason for the block, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/663004/4/modules/varnish/templates/upload-frontend.inc.vcl.erb#380, so that whoever is responsible for this can figure out how to reach us.

Small - and happy! - status update: we managed to get in touch with the app developers, who confirmed our suspicions were correct, and promised a quick turnaround in posting an updated version of the application. We will thus hold back the banning of the url for now, awaiting for confirmation of the desired effect to reduce the potential harmful impact on the application users.

Given how much "sample code" we found around the internet using that url, it might still be a good idea to merge the patch later just to prevent this from happening again.

The sort of suggests the developer community needs an example image to use in example code, the way we do for urls (e.g. http://example.com ).

Although I assume hosting it ourselves would be out of scope. (We actually do have a category for these on commons though! https://commons.wikimedia.org/wiki/Example_images )

Dzahn added a comment.Fri, Feb 12, 4:37 PM

The sort of suggests the developer community needs an example image to use in example code, the way we do for urls (e.g. http://example.com ).
Although I assume hosting it ourselves would be out of scope.

There is https://placeholder.com/ which seems to position itself as just that, placeholder images for designers.

Harej awarded a token.Sat, Feb 13, 7:00 PM
Harej added a subscriber: Harej.

Quick fly-by, but this image corresponds to an internet meme in Hong Kong, though seems unlikely it affects EQSIN servers.

I suggest it to be similar to a placeholder using this image in large sites, it may be server side.

Need more info on UAs before some conclusion can be made.

The UA in question used an empty user agent string. See also User-Agent policy, especially the last paragraph about as-needed enforcement.

Zardula removed a subscriber: Zardula.Sun, Feb 14, 7:31 AM
Joe closed this task as Resolved.Wed, Feb 17, 10:11 AM
Joe claimed this task.

Given the original issue is definitely resolved, there is no point in keeping this task open. Kudos to the app developers who managed to fix this issue promptly!

Asartea removed a subscriber: Asartea.Wed, Feb 17, 10:17 AM