Changing status as system users can now have private data access thanks to Andrew.

Awesome!!! Thank you so much, @Ottomata! :D

Okie dokie, here's where I ended up:

Dmitry brought up some good points on using a summary event approach instead so I'll redesign the schema. @Sharvaniharan: I'll ping you when it's ready. I'm sorry you'll have to reinstrument.

In T190931#4258916, @Sharvaniharan wrote:

In T190931#4254838, @mpopov wrote:

• In case of action = 'search':
  • extras is "<cancelled>" if user backs out without typing anything in the search bar
  • extras is the search string if user searches for a language but doesn't add and just backs out (e.g. "Spanglish")
  • extras is "<added> if user concludes interaction with searching for languages by adding a language they found

Just to be clear here... we are only mentioning 'added', and not which language was added?

Sounds good, @Ottomata! Updated and I'll keep this in mind going forward.

In T191859#4254905, @Ottomata wrote:

ts field for client-side timestamps in case the device goes offline and the event is queued up for a future opportunity

Orrrrr maybe dt in ISO-8601 ? :D

Thanks for the excellent feedback @Sharvaniharan & @RHo! Additional recommendations per our discussion:

My team was in disarray and restructuring when bulk of the work happened on this (hence the delayed responses and lack of CR) and now we don't have the need or the bandwidth for this. We will continue testing changes locally as we have been doing for years, although these days we're not even actively working on any repos/packages listed.

In T191859#4178146, @Tbayer wrote:

The reason to hash app_install_id is because these events would end up somewhere where we would be able to join with behavioral data sent by mobile apps, which we DON'T want

To clarify just in case, it's fine to log app_install_id in connection with user actions, it has been done in many different schemas for years. And "behavioral data" would seem to describe this data here too.
So I guess the "don't want" here refers to connecting users IDs with those other schemas via the app install ID, right? (in which case, fully agreed, although it seems we had been trying to prevent that with Method 1 or Method 2 anyway)

Done for Android.

Having reviewed this, I have the following recommendations:

In T186768#4228658, @NHarateh_WMF wrote:

@mpopov

date, time in UTC, timezone (including daylight savings) offset from UTC

ISO 8601 specifies that it’s the local time with the offset to UTC - is that acceptable?

Just uploaded the data to Go Fish Digital.

I suppose this is as done as can be until we hire a manager to flesh out the page some more.

Cancelling this request as I will be uploading the data to them instead.

In T194961#4221017, @LGoto wrote:

Hi @mpopov Is this for you?

@EBjune @RobH: BTW @JKatzWMF and I are going to be changing which properties are tracked in GSC (namely getting rid of all but 1-2 HTTP variants in favor of HTTPS) as well as which project classes and languages are tracked. We expect to be done with this "sprint cleaning" endeavor sometime in the next two weeks so I'll post an update here when it's ready.

1. Part 2

Declining for now but open to re-opening in the future if the parent task is re-opened and this work is needed.

@debt: closing for now but if you think this is work that needs to be done feel free to re-open

@debt: closing for now but if you think this is work that needs to be done feel free to re-open

Pinging @RStallman-legalteam & @JbuattiWMF to confirm that Go Fish Digital have signed the NDAs so that Ops can proceed with adding their public SSH key to the list of allowed keys.

In T193694#4180609, @jmatazzoni wrote:

Thanks @mpopov! As you note, not all current mapframe wikipedias are accounted for in your first stats page (missing are Arabic, Bulgarian, Czech, Spanish, Kannada, Latvian, Portuguese, English).

But here is the bigger issue: we are about to release mapframe to 277 more Wikipedias—essentially all wikipedias except nine flagged revision wikis. We need to be able to track usage on these as well. What do you suggest? How should they be added in?

Will your stats page be able to scale up to measure hundreds more? Is the general "Wikipedia" figure already accounting for all wikipedias programmatically, or does it just add up the 11 you list on the page? What about the spreadsheet: what will happen if we start loading hundreds of wikis? Should we pick some representative wikis we want to measure?

@Catrope: I updated the repository with instructions: https://github.com/wikimedia-research/Discovery-Interactive-Adhoc-Usage#re-run-instructions

Sooooo…most of this has actually already been done. We have per-wiki daily stats beginning on 2017-09-14 over at:

@Deskana: Progress update: I have 4 days of data (~12GB gzipped) and right now I have a script that's verifying ~20K IP addresses to determine which ones are legit and which ones spoofed the UA and pretended to be one of those crawlers. As you might expect, that part is taking some time.

In T184092#3998152, @mpopov wrote:

@Charlotte: since the user can switch between modes multiple times in any time period, are we interested in (1) % of users who have tried out the two modes† or (2) at a particular snapshot in time, what's the breakdown of people using each theme?

@Charlotte: did you mean to make this iOS ticket part of the Android baseline analytics epic? Also, iOS team is long way away from having data that can be presented; should we replace the parent task with T192819?

@debt: same as with T183028 can you please update the description if this is still something that will need to be worked on? It looks like this is supposed to happen after T176428 (if that ever happens?) but that's not an A/B test, so what is the requested deliverable?

@debt: can you please update the description if this is still something that will need to be worked on? It looks like this is supposed to happen after T178006 (if that ever happens?) but that's not an A/B test, so what is the requested deliverable?

In T193052#4159509, @chelsyx wrote:

FAQs of Baiduspider (in English, include UA): http://help.baidu.com/question?prod_id=99&class=0&id=3001
How to identify Baiduspider (in Chinese, let me know if you can't understand it with google translate): https://ziyuan.baidu.com/college/articleinfo?id=1002

@chelsyx Can you please help me? I’ve been able to find documentation (UserAgent strings and instructions for verifying) on Google’s, Bing’s, and Yandex’s crawlers but the best I’ve been able to find for Baidu (in English) is this blog post by a third party from 2011: https://chineseseoshifu.com/blog/new-baidu-user-agent-baiduspider.html so I suspect any official documentation about what Baiduspider’s UA looks like these days (or how to verify) would be on Baidu's website and in Chinese.

@Deskana: Googlebot Mobile does not seem to be a thing anymore according to its lack of presence on https://support.google.com/webmasters/answer/1061943

In T191859#4156112, @Tbayer wrote:

Method 1 has the disadvantage that we would be able to find out username given crossDeviceID, which is not the case for Method 2.

How is that not the case for Method 2?

In T191859#4155885, @Nuria wrote:

@mpopov FYI that adding things to X-Analytics does not work automagically, we strongly recommend team to track interactions using events.
Want to clarify With @chelsyx and @Fjalapeno that events do not have to be send from client side, the server can also send them. EL has a mediawiki server side client too.

@APalmer_WMF @Fjalapeno @Jhernandez: can y'all please take a look at the updated description and let us know if you have any questions or concerns.

Done

In T191859#4152718, @Tgr wrote:

All these new proposals sound a bit overcomplicated. Why not just use X-Analytics? There is already a purge mechanism for raw webrequest data, right?

0.7.0 on CRAN as of 2018-03-21. Re-installed the package on prod & beta

Majority of this is done. Deb can ping Chelsy or me if she has any more questions or issues accessing the data.