One possible cross check we can do here is to make sure our data represents (percentage wise) our android users (for metrics that are available in all versions such as session length). So if we look at android data per OS and say we have 20% of users on android 5 our data collection on the EL end for sessions (regardless of sampling) should have also 20% users on android 5. If sampling however is heavily biased towards some app versions/OS we might have found our issue.

In T186682#3953874, @Nuria wrote:

One possible cross check we can do here is to make sure our data represents (percentage wise) our android users for metrics susch as session legth. So if we look at android data per OS and say we have 20% of users on android 5 our data collection on the EL end for sessions (regardless of sampling) should have also 20% users on android 5. If sampling however is heavily biased towards some app versions/OS we might have found our issue.

Thanks @Nuria !

On 2018-2-1, in MobileWikiAppSessions, there are 48,729 production app and 45,433 beta app. In webrequest, there are 900,565 production app and 50,913 beta app. The beta sampling rate is closed to 1:1, but the production sampling rate is around 1:18.5.

I broke down the counts by version type (beta vs production) and os. The proportions below don't seem very different. (Since there are too many os, I only show the top 15 here.)

For production, we saw large discrepancies in Android 7.0 and 6.0 (The table only shows top 15):

Then I broke down the counts by app version. The tables only show top 15 versions. And we saw large discrepancies in beta versions, but not in production versions:

Query:

SELECT 
IF(user_agent_map['wmf_app_version'] LIKE '%-r-%', 'prod', 'beta') AS app_version,
CONCAT(user_agent_map['os_family'], '_', user_agent_map['os_major'], '_', user_agent_map['os_minor']) AS os,
COUNT(DISTINCT IF(x_analytics_map['wmfuuid'] IS NOT NULL, x_analytics_map['wmfuuid'], PARSE_URL(CONCAT('http://', uri_host, uri_path, uri_query), 'QUERY', 'appInstallID'))) AS n_users
FROM wmf.webrequest
WHERE year=2018 AND month=2 AND day =1
AND http_status IN('200', '304')
AND user_agent_map['os_family'] = 'Android'
AND access_method = 'mobile app'
AND IF(x_analytics_map['wmfuuid'] IS NOT NULL, x_analytics_map['wmfuuid'], PARSE_URL(CONCAT('http://', uri_host, uri_path, uri_query), 'QUERY', 'appInstallID')) IS NOT NULL
GROUP BY IF(user_agent_map['wmf_app_version'] LIKE '%-r-%', 'prod', 'beta'), user_agent_map['os_family'], user_agent_map['os_major'], user_agent_map['os_minor'];

SELECT
IF(useragent.wmf_app_version LIKE '%-r-%', 'prod', 'beta') AS app_version,
CONCAT(useragent.os_family, '_', useragent.os_major, '_', useragent.os_minor) AS os,
COUNT(DISTINCT event.appInstallID) AS n_users
FROM event.mobilewikiappsessions
WHERE useragent.os_family = 'Android'
AND year=2018 AND month=2 AND day=1
GROUP BY IF(useragent.wmf_app_version LIKE '%-r-%', 'prod', 'beta'), useragent.os_family, useragent.os_major, useragent.os_minor;

@Dbrant Does this breakdown ring any bell?

@Charlotte, according to the breakdown tables above, some Android os/versions are over-represented and some are under-represented in our Eventlogging tables -- MobileWikiAppSessions and possibly others as well. This means the data we collected is not representative of the whole population of our users.

I suggest we set the priority of this ticket to high and move it to the bug backlog, since this bug and T186768 may make the analysis in T184098 (and it's sub-tickets) unreliable.

Loads of work went into this, @chelsyx
Nice work.

If time intervals are the same on webrequest and eventlogging this data (2018/02/01) indeed does not match what we would expect to see for features like sessions that are going to be sampled across the user base. things that come to mind (thinking only about prod versions):

The sampling 1/100 does not seem to apply in any case. The ratio of EL(uniques)/webrequest(uniques) 56,061/ 1,101,078~ 5% is constant across OS and app versions if you total your 15 records. So we are sampling 4 times what we think we are, right?

While not the best situation, this might not be a problem if - like you said - at the beginning data is sampled across the user base but that does not seem to be the case in the case of OS.

Proportions seem OK when you look at app versions by themselves, each version has a similar proportion of data in EL and webrequest for the day in which you run your calculations so percentage of uniques per app version matches across the two datasources. I just do not understand how with equal numbers per version we can get such a different numbers per OS.

Well, this is going to be the biggest facepalm in history:

Upon closer scrutiny of our code, while the intention was for the sampling to apply on a per-user basis, the actual effect turns out not to be the case. In fact, the sampling instead effectively applies to each instance of launching the app. Meaning that a single user might be in the sampling bucket during one usage of the app, but then might no longer be in the bucket the next time the app is launched.

I'm guessing this would invalidate any attempts at measuring unique users from this particular schema. Sorry for all the confusion! :(

We'll be sure to correct this inconsistency in our next update.

Thank you very much @Dbrant !

Given that the sampling applies to each instance of launching the app, if a user launched the app and get sampled to be tracked by EL, then the user closed the app for 20 min (less than the 30 min inactivity in our definition of a session), and then the user launched the app again but didn't get sampled this time. In this situation, only the interaction with the app before the 20 mins' break got logged by MobileWikiAppSessions, is that correct?

In the case you describe, there wouldn't be anything logged at all, since the session is still continuing -- not enough time has elapsed for one session to expire and the next one to begin. (this is independent of actually logging the session to EL)

If the user launches the app, and the current instance of the app happens to be in the sampling bucket, and the previous session has expired, then the previous session will be logged.

Change 409450 had a related patch set uploaded (by Dbrant; owner: Dbrant):
[apps/android/wikipedia@master] WIP: Sample eventlogging schemas based on appInstallID, not current instance.

https://gerrit.wikimedia.org/r/409450

In fact, the sampling instead effectively applies to each instance of launching the app

Ok, at least we know now!
This means that users that use the app most are sampled more frequently which means that data is not random on the user/ phone dimension, and that we cannnot use it to determine the number of unique users (as you mentioned)
Now, sampling per session is not too bad, this is how data is sampled on our desktop site for the most part.

This still does not explain the low number of android 7 so we might have another problem besides this one.

Given these findings, I don't think the OS breakdown data can be trusted at this point, either. The way the current (incorrect) code works, the app can be placed into a sampling bucket every time it's started. However, in the world of Android, "starting" the app is not a predictable concept. Even if all of the app's windows are closed, the app's instance can still be kept running in the background by the system.

And of course, some versions of Android might be more greedy about garbage-collecting inactive apps, whereas others might allow app instances to persist for a long time. It's entirely possible that Android 7.0 is less greedy about terminating inactive apps, which means that there's less churn through re-bucketing of app instances, which might lead to the reduced EL numbers that we see.

I think we should wait and re-run the OS numbers after we deploy the update that fixes the per-user nature of the session schema.

I see, this means that we are going to have to throw away a couple years of data unless we can resample it somehow (random sampling from what we have? cc @chelsey)

Going forward let's please make a practice for developers to do basic vetting of metrics. Example: notice that in this case to see the sampling oddities it was enough to add the uniques in both sources (a simple addition, no stats). Let's not fire and forget metrics but rather follow through a bit to make sure things add up.

In T186682#3964824, @Nuria wrote:

Going forward let's please make a practice for developers to do basic vetting of metrics. Example: notice that in this case to see the sampling oddities it was enough to add the uniques in both sources (a simple addition, no stats). Let's not fire and forget metrics but rather follow through a bit to make sure things add up.

I agree in general (people should double check after deployment as an essential practice) but do want to bring up that the modern app teams do do that in my experience. The buggy code is from almost 3 years ago (from https://phabricator.wikimedia.org/rAPAW8c39a9a96874ba537713cd8325ef5c9af2bd915f I think) and in that time people have learned and improved, so it's kind of unfair to @Dbrant et al. to point at this discovery of a nearly 3 year old bug as if it's a recent mistake and we should be better going forward.

No finger pointing on my end, at all. I am focused on looking for solutions and better process going forward, not guilty parties. If vetting is happening already, sounds excellent.

Change 409450 merged by jenkins-bot:
[apps/android/wikipedia@master] Sample eventlogging schemas based on appInstallID, not current instance.

https://gerrit.wikimedia.org/r/409450

In T186682#3964824, @Nuria wrote:

I see, this means that we are going to have to throw away a couple years of data unless we can resample it somehow (random sampling from what we have? cc @chelsey)

Hi @Nuria, sorry I'm late. Random re-sampling may or may not be helpful, depending on whether the interested metric is measured on the user level. For example, if we want to compute daily avg pageviews per user using MobileWikiAppSessions, we can't get that from the old data because it's very unlikely that every sessions of any particular user in a day are selected to be logged; however, if I want to compute how much of the page users scrolls through on average using MobileWikiAppPageScroll, which use the same sampling method as MobileWikiAppSessions, I can re-sample from the current data to avoid certain OS being overrepresented in the dataset.

In T186682#3959981, @Dbrant wrote:

And of course, some versions of Android might be more greedy about garbage-collecting inactive apps, whereas others might allow app instances to persist for a long time. It's entirely possible that Android 7.0 is less greedy about terminating inactive apps, which means that there's less churn through re-bucketing of app instances, which might lead to the reduced EL numbers that we see.

@Dbrant Does this mean that aggregating session length across OS is not meaningful (e.g. daily average session length) because session length is correlated with OS (e.g. Android 7.0 are more likely to have a longer session)? Maybe we should compare session length within OS groups?

@chelsyx Agreed, +1

Does this mean that aggregating session length across OS is not meaningful (e.g. daily average session length) because session length is correlated with OS (e.g. Android 7.0 are more likely to have a longer session)

Also , better android version means newer device and likely a better connection and better connections are always associated with longer sessions in mobile (not just for wikipedia but in general).

@Dbrant: which release of the app will have the fix?