Page MenuHomePhabricator
Create Task
Maniphest T132969

Wikipedia app hits loads.php on bits.wikimedia.org
Closed, ResolvedPublic
Actions

Description

While inspecting varnish_text traffic, I noticed quite a few requests to bits.wikimedia.org flying by. Aside from the usual third-party favicon pointers and geoiplookup abusers, there were lots of requests coming from a user-agents similar to "WikipediaApp/2.0 * (Android *) Google Play".

I collected about a half dozen samples of 100,000 consecutive requests (represents ~ 7 seconds of traffic). Each sample about 1500-2000 requests are for bits.wikimedia.org, with about 100 of them from WikipediaApp hitting various wiki's load.php.

It doesn't look like an isolated case as it's well distributed (multiple user agent variations, different IPs, through different edge sites).

Filtered example:

{
  "uri_host": "bits.wikimedia.org",
  "uri_path": "/it.wikipedia.org/load.php",
  "uri_query": "?debug=false&lang=en&modules=mobile.app.preview&only=styles&skin=vector",
  "user_agent": "WikipediaApp/2.0-r-2014-08-* (Android *; Phone) Google Play",
  "cache_status": "hit",
  "content_type": "text/css; charset=utf-8",
}

These same user agents are also logging events to /event.gif on bits.wikimedia.org.

Related Objects
Search...

Event Timeline

Krinkle created this task.Apr 18 2016, 9:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 18 2016, 9:05 PM
Krinkle added a parent task: T107430: Decom bits.wikimedia.org hostname.Apr 18 2016, 9:06 PM
Krinkle added projects: Traffic, SRE, Mobile-Apps.
Krinkle updated the task description. (Show Details)
Krenair subscribed.EditedApr 18 2016, 9:17 PM

2014-08-*? Those are quite old versions. Did you get anything more recent than May 2015 (https://gerrit.wikimedia.org/r/#/c/208315/) ?
Only remaining reference I found to bits in apps/android/wikipedia.git was #PREFIX="http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org" in scripts/make-css-assets.bash

Krinkle added a comment.Apr 18 2016, 10:06 PM

A few more (varnish_text, bits.wikimedia.org, WikipediaApp):

10 WikipediaApp/2.0 2014-08 Android
 2 WikipediaApp/2.0 2014-09 Android
 3 WikipediaApp/2.0 2014-10 Android
 6 WikipediaApp/2.0 2014-11 Android
15 WikipediaApp/2.0 2014-12 Android
33 WikipediaApp/2.0 2015-01 Android
 4 WikipediaApp/2.0 2015-03 Android
 5 WikipediaApp/2.0 2015-04 Android
 2 WikipediaApp/4.0.3  iPhone
 4 WikipediaApp/4.0.6  iPhone

This is from about 1 minute of unsampled Varnish traffic.

$ kafkacat -b kafka1012.eqiad.wmnet,kafka1013.eqiad.wmnet,kafka1014.eqiad.wmnet,kafka1018.eqiad.wmnet,kafka1020.eqiad.wmnet,kafka1022.eqiad.wmnet -C -t webrequest_text -c 100000 | grep -i bits.wikimedia.org | grep WikipediaApp | jq -r .user_agent > results.txt
$ cat results.txt | sed -r 's/(WikipediaApp\/[0-9.]*)(-[a-z]*-)?([0-9]*-[0-9]*)?[^\(]*\(*([A-Za-z]*).*/\1 \3 \4/' | sort | uniq -c
fgiunchedi triaged this task as Medium priority.Apr 27 2016, 2:35 PM
• Mholloway added a project: Wikipedia-Android-App-Backlog.Jun 3 2016, 1:49 AM
• Mholloway subscribed.
• Mholloway added a subscriber: • Fjalapeno.Jun 4 2016, 3:05 AM

I looked over our past usage of bits.wikimedia.org, and found that we used it for (1) automatically downloading style updates (we no longer do this) and config settings (now moved to meta), and for event logging (ditto). In each case, if the request fails, an exception will be thrown and caught and there shouldn't be any negative consequence for the user.

I see in @Krinkle's sample that at least a handful of the requests are coming from iOS app UAs, so I'll ping @Fjalapeno to confirm for iOS as well, but as far as the Android app goes, these requests from legacy app versions shouldn't pose a problem for decommissioning bits.wikimedia.org.

Dbrant renamed this task from WikipediaApp for Android hits loads.php on bits.wikimedia.org to Wikipedia app hits loads.php on bits.wikimedia.org.Jun 8 2016, 7:51 PM
Dbrant edited projects, added: Wikipedia-iOS-App-Backlog; removed: Wikipedia-Android-App-Backlog.
• JMinor raised the priority of this task from Medium to Needs Triage.Jun 14 2016, 6:35 AM
• JMinor moved this task from Needs Triage to Engineering Backlog on the Wikipedia-iOS-App-Backlog board.
Krinkle added a comment.Jul 4 2016, 5:53 PM

I filtered another 1,000,000 live requests (~1 minute capture). As before, it's primarily from Android. Previously it matched 84 reqs. This time 143:

31 WikipediaApp/2.0 2014-08 Android
12 WikipediaApp/2.0 2014-11 Android
23 WikipediaApp/2.0 2014-12 Android
44 WikipediaApp/2.0 2015-01 Android
16 WikipediaApp/2.0 2015-03 Android
 5 WikipediaApp/2.0 2015-04 Android
 2 WikipediaApp/2.0  Android
 6 WikipediaApp/4.1.0  iPhone
 4 WikipediaApp/4.1.2  iPhone
$ kafkacat -b kafka1012.eqiad.wmnet,kafka1013.eqiad.wmnet,kafka1014.eqiad.wmnet,kafka1018.eqiad.wmnet,kafka1020.eqiad.wmnet,kafka1022.eqiad.wmnet -C -t webrequest_text -c 1000000 | grep -i bits.wikimedia.org | grep WikipediaApp | jq -r .user_agent > results.txt
$ cat results.txt | sed -r 's/(WikipediaApp\/[0-9.]*)(-[a-z]*-)?([0-9]*-[0-9]*)?[^\(]*\(*([A-Za-z]*).*/\1 \3 \4/' | sort | uniq -c
• Fjalapeno added a comment.Jul 5 2016, 2:59 PM

@Krinkle the version of the iOS app that made those requests is a legacy version - the iOS app no longer makes calls to bits.wikimedia.org. Feel free to decommission.

• Fjalapeno moved this task from Engineering Backlog to Tracking on the Wikipedia-iOS-App-Backlog board.Jul 5 2016, 3:00 PM
MoritzMuehlenhoff triaged this task as Medium priority.Jul 8 2016, 10:10 AM
BBlack subscribed.Aug 3 2016, 9:06 PM
In T132969#2429719, @Fjalapeno wrote:

@Krinkle the version of the iOS app that made those requests is a legacy version - the iOS app no longer makes calls to bits.wikimedia.org. Feel free to decommission.

What about the Android ones from 2015-04 and older? Do we have some kind of policy on supporting un-upgraded versions?

• Mholloway added a subscriber: Dbrant.Aug 3 2016, 9:18 PM

@BBlack, there's no policy on supporting un-upgraded versions of which I'm aware (but I'll add @Dbrant as product owner here for comment).

In any case, it's OK if these requests fail.

BBlack closed this task as Resolved.Aug 18 2016, 4:43 PM
BBlack claimed this task.

With no movement for a couple of weeks here and the various above comments (only outdated app versions, ok for requests to fail, etc), I'm going to consider this effectively resolved

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits