Extract the UUID and timestamp from all pageviews within an N-day period for a sample of App UUIDs

What is the timeperiod exactly?

What is the frecuency of this report? daily? weeekly? (seems than more than weekly it will be too much)

Good q: Deskana?

The high level requirement is that the data be comparable with the old ad-hoc reports. So whatever time period was used for the ad-hoc reports, it should be reused for the automated reports. Was it a backwards looking 30 day window?

In term of frequency of the reports, weekly is fine. Monthly would be too infrequent. Daily would be nice to have, but is by no means necessary.

Daily seems that for a global user base wouldn't work so well, anything lower than weekly is not likely to be very meaningful (Oliver correct me if I am wrong)

@DarTar: Could you elaborate a bit on why do we need a geometric mean for this data, what does it represent? Thanks.

I spoke to Ironholds on IRC:

Run the report weekly on a 30 day window
30 days looking backward is what the previous ad hoc reports used

Geometric Mean is required

for example you get 10 400-second sessions and 1 40,000-second
mean is 4,000 except nobody, zero people, had a mean session length of 4,000
a mean points us to a place on the density curve where nobody lives.
The distribution of session length and several other variables is, at best, log-normal.
In this example the geometric mean is ~600 seconds

BTW the reports Oliver generated are here:
http://datasets.wikimedia.org/aggregate-datasets/apps/

The new automated report will append data to these files.

@Deskana we're assuming you don't need this data backfilled. We couldn't anyway, the cluster only has 60 days of rolling data and none of that has the fields we need yet (see blocking tasks).

In T86535#1143232, @kevinator wrote:

BTW the reports Oliver generated are here:
http://datasets.wikimedia.org/aggregate-datasets/apps/

The new automated report will append data to these files.

That's fine, but you can be sure to put some kind of visual indicator when the data stops being generated by the new method and starts being generated by the new method, so that we don't compare incomparable numbers? :-)

In T86535#1143232, @kevinator wrote:

BTW the reports Oliver generated are here:
http://datasets.wikimedia.org/aggregate-datasets/apps/

The new automated report will append data to these files.

And does this mean that the uniques counting will also start to be appended to these files? The current setup is very suboptimal, I have to ssh into stat1002 and try to download the files. I am not good with computer. :-)

And does this mean that the uniques counting will also start to be appended to these files? The current setup is very suboptimal, I have to ssh into stat1002 and try to download the files. I am not good with computer. :-)

Not for now, sorry. And .....I will confirm with kevin cause I am not sure was not our plan to append to these files either for session data.

Old uniques data should be deleted as we know in some cases is 20% incorrect.

In T86535#1143522, @Nuria wrote:

Old uniques data should be deleted as we know in some cases is 20% incorrect.

Please do not do that. The mobile apps team relies on this data for its quarterly review.

Please do not do that. The mobile apps team relies on this data for its quarterly review.

Got it. Please be aware of the precision of one dataset and the other one.

In T86535#1143572, @Nuria wrote:

Please do not do that. The mobile apps team relies on this data for its quarterly review.

Got it. Please be aware of the precision of one dataset and the other one.

Absolutely! I'll be sure to point out that the error makes the numbers generated by the two methodologies not be directly comparable. Thanks.

Change 199935 had a related patch set uploaded (by Mforns):
[WIP] Add Apps session metrics job

https://gerrit.wikimedia.org/r/199935

I am not sure how the geometric calculation was done before as it seems like it's too big of a number and needs a numerical approximation (it doesn't seem you could multiply sessions lengths for as many sessions as we have and hope you still get a number that can fit in a given type).

We will include quantile calculations for the median but not geometric mean.

Ball is on mobile team's court to document uuid field here: https://wikitech.wikimedia.org/wiki/X-Analytics

Is is uuid or wmfuuid?

Documentation above has been added. it's wmfuuid.

remaining work:

• schedule task in oozie
• store in file similarly to how it was done here: https://hue.wikimedia.org/filebrowser/view/wmf/data/archive/mobile_apps/uniques/

Change 199935 merged by Joal:
Add Apps session metrics job

https://gerrit.wikimedia.org/r/199935

Epilogue: It occurred to me that (besides this Phabricator task and the published code) this table was never publicly documented. I have started a page here, feel free to edit: https://wikitech.wikimedia.org/wiki/Analytics/Data/mobile_apps_session_metrics