Some sources of information to look at:
https://addshore.com/2018/02/wmde-event-logging-intro/
https://www.mediawiki.org/wiki/Extension:EventLogging/Guide

I confirmed with WMF Analytics that we can package these queries as a directory in [[ https://gerrit.wikimedia.org/r/#/admin/projects/analytics/reportupdater-queries | reportupdater-queries ]], which will update TSV reports with daily granularity.

Note that I'm taking the phrase "summary statistics" from the task description and running with it. I'm avoiding the grafana dashboard because I think we can get what we need with far less complexity.

Change 542419 had a related patch set uploaded (by Awight; owner: Awight):
[analytics/reportupdater-queries@master] New report for Reference Previews

https://gerrit.wikimedia.org/r/542419

I learned that nobody else exports from Reportupdater to Graphite, they visualize using Dashiki instead (see below). Consider implementing our graphs there.

15:09 < nuria> awight: the other way would be dashiki  for things like: https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os
15:10 < nuria> awight:  reportupdater is mostly visualized with dashiki:  
               https://language-reportcard.wmflabs.org/#projects=ptwiki,idwiki,eswiki,viwiki,ukwiki,dewiki,trwiki,ruwiki,frwiki/metrics=Content%20Translation
15:10 < nuria> awight: for public dashboards

Started playing with the first bit of data, here: https://grafana.wikimedia.org/d/aHQQ_20Wz/awight-reference-previews?orgId=1&from=now-7d&to=now

Change 542419 merged by Mforns:
[analytics/reportupdater-queries@master] New reports for Reference Previews

https://gerrit.wikimedia.org/r/542419

Change 547715 had a related patch set uploaded (by Awight; owner: Awight):
[operations/puppet@production] Install a cron job to produce Reference Previews metrics

https://gerrit.wikimedia.org/r/547715

Change 547715 merged by Elukey:
[operations/puppet@production] report updater job: produce Reference Previews metrics

https://gerrit.wikimedia.org/r/547715

I'm debugging why baseline data isn't landing in Grafana yet. We're averaging c. 1.5 events/second on the baseline topic, which should come out to about 140k events per day. That was what we were seeing until Nov 1st, at which point the counts dropped to a few dozen.

In T233108#5643443, @awight wrote:

I'm debugging why baseline data isn't landing in Grafana yet. We're averaging c. 1.5 events/second on the baseline topic, which should come out to about 140k events per day. That was what we were seeing until Nov 1st, at which point the counts dropped to a few dozen.

Oh. It's something simple in my query, because I see plenty of data in Hadoop.

Change 549437 had a related patch set uploaded (by Awight; owner: Awight):
[analytics/reportupdater-queries@master] Fix another nonexistent field

https://gerrit.wikimedia.org/r/549437

Change 549437 merged by Mforns:
[analytics/reportupdater-queries@master] Fix another nonexistent field

https://gerrit.wikimedia.org/r/549437

Draft dashboard can be found at https://grafana.wikimedia.org/d/aHQQ_20Wz/reference-previews-usage?orgId=1

The dashboard is ready for review. One think I feel uneasy about was my decision to calculate the references viewed for RP like this:

references views per pageview = reference popups rendered per pageview + footnote clicks per pageview.

The two terms on the right-hand side are rates calculated from different sources, so adding them together does ugly things to the error margins and has other mathematical implications that I don't fully understand.

Another thing that bothers me is that the pageview count is dramatically different between the Cite and Popups metrics streams, by a factor of about 2.5x as many pageviews recorded from Popups, across various wikis. This was unexpected, I would have thought the ratio would be close to 1:1 on wikis with Popups enabled, and overall Popups would be lower because many wikis don't have it enabled. My assumptions were wrong, and I haven't been able to explain the difference. Maybe the Popups tracking code is executed multiple times, in which case our "per pageview" stats are much lower than they should be.

This is concerning enough that I need to reopen the child task.

Let's demo and close.

Just re-adding tags that got accidentally removed.

Review meeting notes:

• "Footnote clicks" needs to be documented.
• Would be nice to dig up the motivations behind the criteria in T214493: Track interaction with ReferencePreviews, and whether we are now able to evaluate our impact, and decide if feature can be made default.
• How can we evaluate our ReferencePreviews impact vs. the Reference Tooltips gadget? Maybe we need to collect more data, but this will be hard to do where the gadget is enabled by default?