Page MenuHomePhabricator
Create Task
Maniphest T229068

App requests classified as pageviews that probably should not be so
Closed, ResolvedPublic
Actions

Description

There are quite a few request from apps being classified as pageviews that *i think* are bot traffic. For example these are all requests fired by the same device in close succession:

8b5b8d8871            |2019-07-01T14:27:58Z|/api/rest_v1/page/mobile-sections-lead/United_States_Secretary_of_Homeland_Security|WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:04Z|/api/rest_v1/page/mobile-sections-lead/List_of_Roman_emperors                      |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:06Z|/api/rest_v1/page/mobile-sections-lead/United_States_Navy_ships                    |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:07Z|/api/rest_v1/page/mobile-sections-lead/John_Ripley_(USMC)                          |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:09Z|/api/rest_v1/page/mobile-sections-lead/Montepulciano_(grape)                       |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:10Z|/api/rest_v1/page/mobile-sections-lead/Barbaresco                                  |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:11Z|/api/rest_v1/page/mobile-sections-lead/Paul_J._Zak                                 |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:12Z|/api/rest_v1/page/mobile-sections-lead/Men's_major_golf_championships              |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:13Z|/api/rest_v1/page/mobile-sections-lead/Alberto_Bettiol                             |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:14Z|/api/rest_v1/page/mobile-sections-lead/George_N._Gillett_Jr.                       |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:15Z|/api/rest_v1/page/mobile-sections-lead/William_Tecumseh_Sherman                    |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:16Z|/api/rest_v1/page/mobile-sections-lead/Coefficient_of_variation                    |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:17Z|/api/rest_v1/page/mobile-sections-lead/Michael_Douglas                             |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:18Z|/api/rest_v1/page/mobile-sections-lead/Barbara_Bush                                |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:18Z|/api/rest_v1/page/mobile-sections-lead/Brunello_di_Montalcino                      |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:19Z|/api/rest_v1/page/mobile-sections-lead/Ron_Howard                                  |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:20Z|/api/rest_v1/page/mobile-sections-lead/Beto_O'Rourke                               |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:21Z|/api/rest_v1/page/mobile-sections-lead/Premier_of_the_People's_Republic_of_China   |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:22Z|/api/rest_v1/page/mobile-sections-lead/FIS_Alpine_Ski_World_Cup                    |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|
|8b5b8d8871            |2019-07-01T14:28:23Z|/api/rest_v1/page/mobile-sections-lead/Lindsey_Vonn                                |WikipediaApp/2.7.225-r-2018-02-06 (Android 9; Phone) Google Play|

Can someone from apps (pinging @Dbrant) take a look at whether these requests might be intentional (app pre-building some kind of content or similar?) In the case of this being intentional traffic it should not be counted as pageviews either but it should not be counted to wards bot activity

Event Timeline

• Nuria created this task.Jul 25 2019, 10:58 PM
Restricted Application added a project: Wikipedia-Android-App-Backlog. · View Herald TranscriptJul 25 2019, 10:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• Charlotte added subscribers: Sharvaniharan, cooltey, • Charlotte.Jul 26 2019, 7:51 AM

Thanks for asking @Nuria. @Dbrant is out on holiday, so @Sharvaniharan or @cooltey might be able to take a look for you.

cooltey added a comment.Jul 26 2019, 6:35 PM

Noticed that the logs were from a very old version 2.7.225-r-2018-02-06

I am guessing that they were because when the app syncs the reading lists and it will make requests to /api/rest_v1/page/mobile-section-lead and /api/rest_v1/page/mobile-sections-remaining for offline reading.

Do you see any recent date log that contains mobile-sections-lead or mobile-sections-remaining, and also do you have the schema of the logs? Thanks!

• Nuria added a comment.Jul 26 2019, 7:03 PM

@cooltey let me make sure i understand your request, the app version might be old but these are recent logs from july 1st.

• Nuria added a comment.Jul 26 2019, 7:13 PM

Also, there are a lot of requests that request the same content over and over which might indicate a problem, see three identical requests in close succession.

004568b0ef2019-07-01T15:00:58Z/api/rest_v1/page/mobile-sections-lead/Mercedes-Benz_CLKWikipediaApp/2.7.50282-r-2019-05-24 (Android 9; Phone) Google Play
004568b0ef2019-07-01T15:00:59Z/api/rest_v1/page/mobile-sections-lead/Mercedes-Benz_CLKWikipediaApp/2.7.50282-r-2019-05-24 (Android 9; Phone) Google Play
004568b0ef2019-07-01T15:01:02Z/api/rest_v1/page/mobile-sections-lead/Mercedes-Benz_CLKWikipediaApp/2.7.50282-r-2019-05-24 (Android 9; Phone) Google Play
004568b0ef2019-07-01T15:01:07Z/api/rest_v1/page/mobile-sections-lead/PumaWikipediaApp/2.7.50282-r-2019-05-24 (Android 9; Phone) Google Play
004568b0ef2019-07-01T15:01:13Z/api/rest_v1/page/mobile-sections-lead/Puma_(Sportartikelhersteller)WikipediaApp/2.7.50282-r-2019-05-24 (Android 9; Phone) Google Play
cooltey added a comment.Jul 26 2019, 9:53 PM

Thanks @Nuria for providing the requests logs from a newer version of the release. (2.7.50282-r-2019-05-24)

I believe someone was using an old version (>1 year ago) of the app and there’s not much we can do, but we can still keep our eyes on it to see if it is still happening in the latest release.

• fdans moved this task from Incoming to Radar on the Analytics board.Jul 29 2019, 3:30 PM
• Charlotte triaged this task as Medium priority.Jul 30 2019, 7:42 PM
• Charlotte edited projects, added Wikipedia-Android-App-Backlog (Android-app-release-v2.7.29x-N-Nanaimo-Bar); removed Wikipedia-Android-App-Backlog.
• Charlotte moved this task from Backlog to Blocked/Waiting on the Wikipedia-Android-App-Backlog (Android-app-release-v2.7.29x-N-Nanaimo-Bar) board.
• Charlotte added a comment.Oct 22 2019, 4:15 PM

Hi @Nuria - is this issue still occurring?

• Nuria added a comment.Oct 22 2019, 9:14 PM

In the 2.7 app, yes, this continues to happen but i cannot see it happen on newer versions.
Also I can see similar pattern in iOS.

Since to calculate this I had to calculate other things i though I'd share that at the 50th percentile users are doing two pageviews on the app per day, the percentile 90 is about 9 pageviews per day.

• Charlotte added a comment.Oct 23 2019, 9:55 AM

Thanks @Nuria - the issues will be resolved once those on the older versions update the app (assuming they do; we don't force updates). Interesting view stats.

• Charlotte closed this task as Resolved.Oct 23 2019, 9:55 AM
Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL