Page MenuHomePhabricator

Drop in iOS app pageviews since version 5.2.0
Closed, ResolvedPublic5 Story Points

Description

The app has been losing most of its pageviews since the rollout of 5.2.0 on September 13 (concurrent with Apple's iOS 10 update):

Looking at the views per version, it seems that the decrease has continued with 5.2.1 (i.e. that it wasn't just a one-time effect caused by a change in 5.2.0):


(from Pivot - access required - ; haven't found out if one can create stacked charts there, so when glancing at this one should keep in one's visual mind that these need to be added up for the total views across all versions)

Event Timeline

Restricted Application added a project: Wikipedia-iOS-App-Backlog. · View Herald TranscriptOct 19 2016, 6:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

FWIW (maybe not too much), this is a bit reminiscent of T132965 for the Android app, which turned out to be related to the handling of the X-Analytics header in connection with RESTBase and the rollout of Mobile Content Service there.

JMinor triaged this task as High priority.Oct 20 2016, 1:19 AM
JMinor added a subscriber: JMinor.

Can we check that our user agent and headers have not regressed?

If the solution is not obvious this will need to wait for the next release.

Current headers:

GET /w/api.php?action=mobileview&format=json&noheadings=true&page=Homicide%3A%20Life%20on%20the%20Street%20%28season%206%29&prop=sections%7Ctext%7Clastmodified%7Clastmodifiedby%7Clanguagecount%7Cid%7Cprotection%7Ceditable%7Cdisplaytitle%7Cthumb%7Cdescription%7Cimage%7Crevision&sectionprop=toclevel%7Cline%7Canchor%7Clevel%7Cnumber%7Cfromtitle%7Cindex&sections=all&thumbwidth=640 HTTP/1.1
Host: en.wikipedia.org
Accept: */*
Cookie: WMF-Last-Access=24-Oct-2016; GeoIP=US:PA:Philadelphia:39.94:-75.15:v4
User-Agent: WikipediaApp/5.3.0.0 (iOS 10.0; Phone)
Accept-Language: en
Accept-Encoding: gzip
Connection: keep-alive

@Tbayer does that user agent string look ok to you? Looking here, that seems to be ok:
https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/PageviewDefinition.java

JMinor added a comment.EditedOct 24 2016, 8:39 PM

For context when this occurred before here was the criteria T131824#2184114

So I ran the regex in the file against our current user agent and it matched

Here is the regex from the Webrequest.java file:

"Wikipedia(App|/5.0.)"

Here is the user agent from 5.3:

WikipediaApp/5.3.0.0 (iOS 10.0; Phone)

Blocking on verification from @Tbayer

JMinor claimed this task.Oct 25 2016, 6:26 PM
JMinor added a subscriber: Fjalapeno.

Hi @madhuvishy and @JAllemandou. We had a similar incident in the spring, and it turned out our user agent had changed. This time, we've checked the user agent against the regex and do not see any issues. Any help or pointers greatly appreciated.

This seems potentially unrelated to app version, but rather OS based. Looks like iOS 10 is not being counted by 9 still works?

Tbayer added a comment.EditedOct 25 2016, 10:25 PM

...

This seems potentially unrelated to app version, but rather OS based. Looks like iOS 10 is not being counted by 9 still works?

It does look like that - almost no (although not zero) views from iOS 10:


(from Pivot, access required)

Tbayer added a comment.EditedOct 25 2016, 10:30 PM

So I ran the regex in the file against our current user agent and it matched

Here is the regex from the Webrequest.java file:

"Wikipedia(App|/5.0.)"

Here is the user agent from 5.3:

WikipediaApp/5.3.0.0 (iOS 10.0; Phone)

As Josh mentioned, the Analytics Engineering team will be best equipped to check if this header works with the existing pageview instrumentation.

However, didn't the UA contain the string "iPhone" too earlier? The error could be either in the logic that determines if the request is classified as is_pageview, or in the classification as an iOS app view. I think the latter is handled by the standard ua-parser library, which one would assume has already been updated upstream for any changes resulting from iOS 10.

Good catch @Tbayer :
The function matching iOS mobile-apps pageviews are expecting "iPhone" in their user agent, and also "Wikipedia/5.0." (that is definitely wrong, flexibility for new versions should be builtin).
I suggest some more analysis confirming those are the only issues impacting the drop and update the code.

Nuria added a subscriber: Nuria.Oct 26 2016, 7:02 PM

@JMinor: Please be so kind to communicate changes to apps user agents to us before they happen so we do not find ourselves in this situation.

Can @Fjalapeno please submit a CR to the pageview definition with the needed changes? We would like for you guys to own the process of driving those changes rather than having to play catch-up on our end. We cannot guarantee backfilling pageviews is possible in all instances.

Nuria added a comment.Nov 1 2016, 6:56 PM

ping @JMinor, ping @Tbayer Please have in mind that we are waiting for a CR from IOS team regarding the proper user agents on definition. We cannot gurantee we can backfill pageviews so the sooner we address the issue the sooner pageviews will be corrected.

Nuria added a comment.Nov 1 2016, 7:04 PM

Please see my comments regarding backfilling when similar UA changes in the app happen earlier this year: https://phabricator.wikimedia.org/T131824

@dr0ptp4kt this seems out of the scope of iOS team engineering development (submitting CR's for analytics code). Is this something the iOS team should own (not the notification of changes, the actual code submissions)?

Nuria added a comment.Nov 1 2016, 8:51 PM

@JMinor: we can own the code submission but not w/o notification of detailed changes in a timely manner before they are deployed to users, as we really do not want to be in the situation of having to backfill pageviews.

Nuria added a comment.Nov 1 2016, 9:00 PM

A sample of OS 10 requests that are not consider pageviews, note tablet versus Phone (not iPhone). @Fjalapeno mentioned some headers above but clearly there is
more variety

{"browser_major":"-","os_family":"iOS","os_major":"10","device_family":"Other","browser_family":"Other","os_minor":"0","wmf_app_version":"5.2.1.942"} false WikipediaApp/5.2.1.942 (iOS 10.0; Phone)
{"browser_major":"-","os_family":"iOS","os_major":"10","device_family":"Other","browser_family":"Other","os_minor":"0","wmf_app_version":"5.2.0.939"} false WikipediaApp/5.2.0.939 (iOS 10.0.2; Tablet)
{"browser_major":"-","os_family":"iOS","os_major":"10","device_family":"Other","browser_family":"Other","os_minor":"0","wmf_app_version":"5.2.0.939"} false WikipediaApp/5.2.0.939 (iOS 10.0.2; Phone)
{"browser_major":"-","os_family":"iOS","os_major":"10","device_family":"Other","browser_family":"Other","os_minor":"0","wmf_app_version":"5.2.1.942"} false WikipediaApp/5.2.1.942 (iOS 10.0.1; Tablet)
{"browser_major":"-","os_family":"iOS","os_major":"10","device_family":"Other","browser_family":"Other","os_minor":"0","wmf_app_version":"5.2.1.942"} false WikipediaApp/5.2.1.942 (iOS 10.0.3; Phone)
{"browser_major":"-","os_family":"iOS","os_major":"10","device_family":"Other","browser_family":"Other","os_minor":"1","wmf_app_version":"5.2.1.942"} false WikipediaApp/5.2.1.942 (iOS 10.1; Tablet)
{"browser_major":"-","os_family":"iOS","os_major":"10","device_family":"Other","browser_family":"Other","os_minor":"0","wmf_app_version":"5.2.1.942"} false WikipediaApp/5.2.1.942 (iOS 10.0.1; Phone)
{"browser_major":"-","os_family":"iOS","os_major":"10","device_family":"Other","browser_family":"Other","os_minor":"0","wmf_app_version":"5.2.1.942"} false WikipediaApp/5.2.1.942 (iOS 10.0.2; Tablet)
{"browser_major":"-","os_family":"iOS","os_major":"10","device_family":"Other","browser_family":"Other","os_minor":"1","wmf_app_version":"5.2.1.942"} false WikipediaApp/5.2.1.942 (iOS 10.1; Phone)
{"browser_major":"-","os_family":"iOS","os_major":"10","device_family":"Other","browser_family":"Other","os_minor":"0","wmf_app_version":"5.2.1.942"} false WikipediaApp/5.2.1.942 (iOS 10.0.2; Phone)

@Nuria Sorry I only provided the phone version of the user agent string. As you pointed out there is a "Tablet" variant.

As to the "iPhone" vs "Phone", I am not sure why "iPhone" is being searched for. According to the file history for this code, we have always used Phone/Tablet:

https://github.com/wikimedia/wikipedia-ios/blob/2b4f10c641a253ddf4d5436883fe1761d16187d2/Wikipedia/Code/WikipediaAppUtils.m#L27

@Nuria can you provide an example of a user agent string from the iOS that was processed correctly?

I know you asked us to communicate changes to you, but I actually don't see any changes in the code and the user agent you posted above look as I would expect. (maybe I am missing something?)

Seeing what worked in the past may shed some light on the issue.

Thanks!

Other notes:
As @JAllemandou pointed out, we should not be looking for hard coded version numbers, so:

Wikipedia(App|/5.0.)

Can be changed to:

WikipediaApp

(Though as far as I can see this wouldn't have caused the regex the fail - and if it did, it would have started failing with version 5.1, not 5.2)

iPhone string is currently necessary in the user_agent for webrequest being considered as pageview, and it looks that new user agents contain iOS only.
Analysis of user_agents in webrequest for this specific case over september and october:

  • User agents containg (iOS or iPhone) and (Wikipedia/5.0 or WikipediaApp)
  • How many of them actually contain iPhone, and how many contain iOS
SELECT
  month,
  day,
 SUM(CASE WHEN (user_agent LIKE '%iPhone%') THEN 1 ELSE 0 END) AS iphone,
 SUM(CASE WHEN (user_agent LIKE '%iOS%') THEN 1 ELSE 0 END) AS iOS
FROM wmf.webrequest
WHERE webrequest_source = 'text'
  AND year = 2016
  AND month IN (9, 10)
  AND (user_agent like '%iOS%' OR user_agent like '%iPhone%')
  AND (user_agent like '%Wikipedia/5.0%' OR user_agent like '%WikipediaApp%')
GROUP BY
  month,
  day
ORDER BY
  month,
  day
LIMIT 1000;

Plotted results:

Nuria added a comment.Nov 2 2016, 8:06 PM

As to the "iPhone" vs "Phone", I am not sure why "iPhone" is being searched for. According to the file history for this code, we have always used Phone/Tablet:

@Fjalapeno To triple check: are sure this also applies to webviews?

Those UAS thus far I think they look something like: {"browser_major":"-","os_family":"iOS","os_major":"9","device_family":"Generic Smartphone","browser_family":"Mobile Safari UIWebView","os_minor":"1","wmf_app_version":"5.2.1.942"} false WikipediaApp/5.2.1.942 (iPhone OS 9.1; Phone)

Nuria set the point value for this task to 5.Nov 3 2016, 3:33 PM

iPhone string is currently necessary in the user_agent for webrequest being considered as pageview, and it looks that new user agents contain iOS only.
Analysis of user_agents in webrequest for this specific case over september and october:

Thanks for checking this! Agree that this recent change in that app's user agent is important and should be documented somewhere. It also affects some other metrics, e.g. the new retention metric where the query used so far had been relying on userAgent LIKE '%iPhone%' as well (T126693#2564106 ).

Nuria moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Nov 10 2016, 8:57 PM
Nuria added a comment.Nov 14 2016, 9:30 PM

Verified that several requests that were not marked as pageviews before are marked as pageviews now. Still @Fjalapeno having documentation as to what constitutes a valid user agent in iOS would be of help. You can add docs to here: https://meta.wikimedia.org/wiki/Research:Page_view

@JoeWalsh @Fjalapeno can one of y'all update the user agent definition/pattern for the analytics team per @Nuria previous comment?

For the record, the change related to Nuria's comment above: https://gerrit.wikimedia.org/r/#/c/319374

Nuria closed this task as Resolved.Nov 29 2016, 7:52 PM

Confirming that the app's registered pageviews appear to have gone back up to normal on or around November 10 (the above change was merged on November 4):


(from Pivot, access required)
CC @Zareenf