Page MenuHomePhabricator

Count requests to RESTBase from the Android app
Closed, ResolvedPublic5 Story Points

Description

In the Android app, add x-analytics request header with value pageview=1 for both RESTBase and MediaWiki php.api endpoints.
(Might as well add the missing preview=1 values to the link preview requests.)

In the analytics-refinery-source the PageviewDefinition for is isAppPageview() should be updated to use the pageview=1 value.

It would be great to get this completed and rolled out to the Android production app before we rollout RESTBase usage to the Android production app.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 2 2016, 5:12 PM
Nuria added a subscriber: Nuria.Mar 2 2016, 5:21 PM

Please use x-analytics header to tag whether a request is a pageview. See for example how is this handled for preview requests on the app:

Value should be pageview=1

https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/PageviewDefinition.java#L195
https://wikitech.wikimedia.org/wiki/X-Analytics

Homework is to make sure that this header passes through restbase and comes into hive

bearND added a subscriber: bearND.Mar 2 2016, 5:39 PM

I think we should do this for both MW API and RESTBase page content requests; either on the lead section or remaining sections requests.

this would certainly simplify the pageview definition if all clients adopted it, so let's do it.

All RB requests pass through the text varnishes, so all accesses (including X-Analytics) are already available as part of the regular log stream. If all text varnish logs are treated equally for page view purposes, then requests with the header set would already be counted.

Nuria added a comment.Mar 2 2016, 7:28 PM

All RB requests pass through the text varnishes, so all accesses (including X-Analytics) are already available as part of the regular log stream.

then we should see those in webrequest table.

@bearND : let us know when you have it implemented it on your end (test app requests are fine) and we can check details. We would also need to update documentation regarding this "pageview=1

bearND claimed this task.Mar 3 2016, 7:14 AM
bearND updated the task description. (Show Details)
bearND moved this task from To Do to Doing on the Mobile-App-Android-Sprint-77-Iridium board.
bearND updated the task description. (Show Details)

Change 274644 had a related patch set uploaded (by BearND):
Use x-analytics pageview to count mobile apps pageviews

https://gerrit.wikimedia.org/r/274644

Change 274645 had a related patch set uploaded (by BearND):
Add X-Analytics request headers for previews and pageviews

https://gerrit.wikimedia.org/r/274645

Milimetric set the point value for this task to 5.Mar 3 2016, 5:45 PM

Change 274645 merged by jenkins-bot:
Add X-Analytics request headers for previews and pageviews

https://gerrit.wikimedia.org/r/274645

@Dbrant If this is really an Analytics task (evidenced by @Milimetric story-pointing it), I would recommend that the Android team subtask for their portion of the work, and stash this task outside of their sprint, to avoid skewing Android's velocity. The subtask can then be independently pointed by Android and added to the sprint in place of this task.

Change 274848 had a related patch set uploaded (by BearND):
Change RB remote config key

https://gerrit.wikimedia.org/r/274848

Nuria claimed this task.Mar 7 2016, 4:01 PM

Change 274848 merged by jenkins-bot:
Change RB remote config key for production app

https://gerrit.wikimedia.org/r/274848

Nuria moved this task from Next Up to In Progress on the Analytics-Kanban board.Mar 8 2016, 9:20 PM

So the 2.1.143 beta release went out last week and (per T128887 and this diff) should be sending requests with "x-analytics: preview=1". However, the webrequest table still does not seem to see them (see below). Is there something else still missing, or is this the wrong query?

hive (default)> select is_pageview, x_analytics_map["preview"] from wmf.webrequest where year=2016 and month=3 AND day = 22 and x_analytics_map["preview"] is not NULL limit 10;
...
OK
is_pageview	_c1
Time taken: 320.668 seconds
Nuria added a comment.Mar 23 2016, 2:23 PM

should be sending requests with "x-analytics: preview=1".

Can android devs confirm (via wireshark or similar) that app is sending the header?

Nuria moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Mar 23 2016, 3:38 PM

Please use x-analytics header to tag whether a request is a pageview

I don't understand, does this mean that now anyone can make any request to the WMF servers be counted as a pageview simply by adding this header?

Nuria added a comment.EditedMar 24 2016, 3:07 PM

I don't understand, does this mean that now anyone can make any request to the WMF servers be counted as a pageview simply by adding this header?

There is a bug on this code though (mea culpa) and that is that we need to first look at filters such us http response code before looking at the tagging. Duh. Thanks for catching this.

The idea behind this change is that for domains that are being counted as "knowledge pageviews" (do not know how else to call them) rather than inspecting urls via regexes on what constitutes a pageviev we are moving towards tagging requests that should be consider pageviews as such.

Nuria moved this task from Done to In Progress on the Analytics-Kanban board.Mar 24 2016, 3:07 PM

Change 279447 had a related patch set uploaded (by Nuria):
Look at filters before looking at the Pageview tagging on x-analytics

https://gerrit.wikimedia.org/r/279447

Nuria moved this task from In Code Review to Done on the Analytics-Kanban board.Apr 5 2016, 4:11 PM
Nuria moved this task from Done to Ready to Deploy on the Analytics-Kanban board.
Nuria closed this task as Resolved.Apr 15 2016, 4:50 PM