Motivation
We want to find out to what extend Reference Previews is improving readers' behavior. This ticket is for getting more info about Reference Previews usage, T231529 is about creating a baseline that we can compare the usage against.
Acceptance Criteria
- How often do people look at a reference pop up relative to the pages being opened? E.g. On average, there were 0.03 showings of a reference pop up per page opened where Reference Previews was deployed.
- In how many percent of the cases do users click on “go to references section” when they see a reference preview?
-
In how many percent of the cases do users click on a link inside the reference itself when they see a reference preview? - How often (absolute number per page being opened) do people (who have referencepreview enabled) click on a link inside the reference itself when they see a reference preview?
- How often (absolute number per page being opened) do people (who have referencepreview enabled) click on a link inside the reference itself, but from the references section
- In how many percent of the cases does an opened reference pop up contain scroll bars?
- Of those times, how often do users scroll? (For now, let’s not separate between horizontally and vertically)
Metrics
We're using a new schema "ReferencePreviewsPopups", which tracks the following actions:
- action: poppedOpen is sent when a reference preview is rendered. scrollbarsPresent: true is set when the reference overflows its bounding box and a vertical scrollbar is present.
- action: scrolled is sent exactly once, if the user scrolls a reference preview either horizontally or vertically.
- action: clickedReferencePreviewsContentLink is sent when a reference content link is clicked in a reference preview.
- action: clickedGoToReferences is sent when "jump to references" is clicked.
This can be supplemented with ReferencePreviewsBaseline to collect all of the metrics we need here, although note that the sampling rate may be different. That will already collect the following:
- action: pageview is recorded when opening a page with Cite enabled. When Reference Previews are enabled for the user, this event will include referencePreviewsEnabled: true.
- action: clickedFootnote when clicking a reference link ("[1]"). Note that this also fires in other circumstances such as when "Jump to references" is clicked, so we need to refine a bit or at least figure out how to calculate the correct statistics. Maybe it's as simple as ignoring this when referencePreviewsEnabled, and using poppedOpen instead.
- action: clickedReferenceContentLink when clicking on a link in the References section.
Previous comments
Conclusion
To summarize how the acceptance criteria map into Grafana dashboards,
| Criterion | Graph |
| How often do people look at a reference pop up relative to the pages being opened? | References viewed, baseline vs. RP |
| In how many percent of the cases do users click on “go to references section” when they see a reference preview? | Popups Go To References Clicks per Pageview |
| In how many percent of the cases do users click on a link inside the reference itself when they see a reference preview? | this seems to be redundant with the next criterion. |
| How often do people who have referencepreview enabled click on a link inside the reference itself when they see a reference preview? | Popups Content Clicks per Pageview |
| How often do people who have referencepreview enabled click on a link inside the reference itself, but from the references section | References Section Content Clicks per Pageview, with Reference Previews |
| In how many percent of the cases does an opened reference pop up contain scroll bars? | Reference Popup has Scrollbars |
| Of those times, how often do users scroll? | Reference Popup is Scrolled when has Scrollbars |