Page MenuHomePhabricator

Track interaction with ReferencePreviews
Open, NormalPublic3 Story Points

Description

Motivation

We want to find out to what extend Reference Previews is improving readers' behavior. This ticket is for getting more info about Reference Previews usage, T231529 is about creating a baseline that we can compare the usage against.

Acceptance Criteria
  • How often do people look at a reference pop up relative to the pages being opened? E.g. On average, there were 0.03 showings of a reference pop up per page opened where Reference Previews was deployed.
  • In how many percent of the cases do users click on “go to references section” when they see a reference preview?
  • In how many percent of the cases do users click on a link inside the reference itself when they see a reference preview?
  • How often (absolute number per page being opened) do people (who have referencepreview enabled) click on a link inside the reference itself when they see a reference preview?
  • How often (absolute number per page being opened) do people (who have referencepreview enabled) click on a link inside the reference itself, but from the references section
  • In how many percent of the cases does an opened reference pop up contain scroll bars?
    • Of those times, how often do users scroll? (For now, let’s not separate between horizontally and vertically)
Metrics

We're using a new schema "ReferencePreviewsPopups", which tracks the following actions:

  • action: poppedOpen is sent when a reference preview is rendered. scrollbarsPresent: true is set when the reference overflows its bounding box and a vertical scrollbar is present.
  • action: scrolled is sent exactly once, if the user scrolls a reference preview either horizontally or vertically.
  • action: clickedReferencePreviewsContentLink is sent when a reference content link is clicked in a reference preview.
  • action: clickedGoToReferences is sent when "jump to references" is clicked.

This can be supplemented with ReferencePreviewsBaseline to collect all of the metrics we need here, although note that the sampling rate may be different. That will already collect the following:

  • action: pageview is recorded when opening a page with Cite enabled. When Reference Previews are enabled for the user, this event will include referencePreviewsEnabled: true.
  • action: clickedFootnote when clicking a reference link ("[1]"). Note that this also fires in other circumstances such as when "Jump to references" is clicked, so we need to refine a bit or at least figure out how to calculate the correct statistics. Maybe it's as simple as ignoring this when referencePreviewsEnabled, and using poppedOpen instead.
  • action: clickedReferenceContentLink when clicking on a link in the References section.

Previous comments

I see the schema has been updated to include ReferenceTooltips. I wonder if it might make sense to use a new schema, given you'll likely want to log reference tooltip previews separately from page previews. I'd imagine reference tooltips will display less often then link previews so if you are leaning heavily on the existing logging code, the sample size for reference tooltips is likely to be very small in comparison to link previews. Does it make sense to "fork" this schema and update the code so that it doesn't log link previews at all, or is data relating to link previews important for the questions you are trying to answer?

Details

Related Gerrit Patches:
mediawiki/extensions/Popups : masterDon't record Popups actions on non-content pages
mediawiki/extensions/Cite : masterTrack pageviews only on content page views, not edits
mediawiki/extensions/Popups : master[WIP]
mediawiki/extensions/Popups : master[WIP] Reduce double-counting of Reference Previews pageviews
mediawiki/extensions/Popups : wmf/1.35.0-wmf.5Ensure that pageviews are recorded at most once
mediawiki/extensions/Popups : masterEnsure that pageviews are recorded at most once
analytics/reportupdater-queries : masterFix nonexistent field in query
analytics/reportupdater-queries : masterNew reports for Reference Previews
analytics/reportupdater-queries : masterReport for ReferencePreviews popups
mediawiki/extensions/Popups : masterRecord pageviews where Reference Previews is enabled
mediawiki/extensions/Cite : masterStop sampling when Reference Previews is enabled
mediawiki/extensions/Popups : masterSample ReferencePreviewsPopups 1:1
mediawiki/extensions/Popups : wmf/1.35.0-wmf.1Tune referencePreviews sampling from 1:1000 to 1:10
mediawiki/extensions/Popups : masterTune referencePreviews sampling from 1:1000 to 1:10
mediawiki/extensions/Popups : masterTune referencePreviews sampling from 1:1000 to 1:10
mediawiki/extensions/Popups : masterTracking for Reference Previews interactions
mediawiki/extensions/Popups : masterCollect metrics for logged-in users as well as anons

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
awight updated the task description. (Show Details)Sep 25 2019, 2:00 PM

Change 539140 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Popups@master] Collect metrics for logged-in users as well as anons

https://gerrit.wikimedia.org/r/539140

awight updated the task description. (Show Details)Sep 27 2019, 9:02 AM
awight updated the task description. (Show Details)
awight updated the task description. (Show Details)
awight updated the task description. (Show Details)
awight updated the task description. (Show Details)Sep 27 2019, 9:15 AM

@GoranSMilovanovic @thiemowmde @Hanna_Petruschat_WMDE
Idea for another metric to compare: How many times the up-arrow is clicked to return from a reference to the article text. If possible, record how often the browser back-button is used for the same purpose. This might give us a nice picture of how we're improving the user experience.

awight updated the task description. (Show Details)Sep 27 2019, 9:26 AM

Change 539140 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] Collect metrics for logged-in users as well as anons

https://gerrit.wikimedia.org/r/539140

Change 538641 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] Tracking for Reference Previews interactions

https://gerrit.wikimedia.org/r/538641

Still no data. I'm thinking we can jump to 1:10 or even 1:1 sampling on Monday, since so few people have this feature enabled.

Change 542324 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Popups@master] Tune referencePreviews sampling from 1:1000 to 1:10

https://gerrit.wikimedia.org/r/542324

Change 542324 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] Tune referencePreviews sampling from 1:1000 to 1:10

https://gerrit.wikimedia.org/r/542324

Change 542326 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Popups@master] Tune referencePreviews sampling from 1:1000 to 1:10

https://gerrit.wikimedia.org/r/542326

Change 542326 abandoned by Awight:
Tune referencePreviews sampling from 1:1000 to 1:10

Reason:
wrong branch

https://gerrit.wikimedia.org/r/542326

Change 542327 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Popups@wmf/1.35.0-wmf.1] Tune referencePreviews sampling from 1:1000 to 1:10

https://gerrit.wikimedia.org/r/542327

Change 542327 abandoned by Awight:
Tune referencePreviews sampling from 1:1000 to 1:10

Reason:
Let's do the train, instead.

https://gerrit.wikimedia.org/r/542327

New plan due to holiday schedules: this sampling rate change will go out with the 1.35.0-wmf2 train, next week.

There are now 15 rows for ReferencePreviewsPopups, all with the "poppedOpen" action. These are all from Oct 21, so we can probably expect this number daily. We should increase sampling again.

Change 545219 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Popups@master] Sample ReferencePreviewsPopups 1:1

https://gerrit.wikimedia.org/r/545219

Change 545219 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] Sample ReferencePreviewsPopups 1:1

https://gerrit.wikimedia.org/r/545219

One of my assumptions was completely off the mark: there are so few referencePreviewsEnabled users that I had to tune the sampling to 1:1. Meanwhile, I was relying on the ReferencePreviewsBaseline tracking (sampled at 1:1000) to give us a few of the most important metrics, such as pageviews. This will probably be statistically unusable, so we need to consider splitting the sampling (e.g. track if inSample(1000) or refPreviewsEnabled) or doing some other dramatic workaround.

Change 545869 had a related patch set uploaded (by Awight; owner: Awight):
[analytics/reportupdater-queries@master] Report for ReferencePreviews popups

https://gerrit.wikimedia.org/r/545869

Change 545870 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Cite@master] Stop sampling for referencePreviewsEnabled

https://gerrit.wikimedia.org/r/545870

Change 545942 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Popups@master] Record pageviews where Reference Previews is enabled

https://gerrit.wikimedia.org/r/545942

Change 545942 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] Record pageviews where Reference Previews is enabled

https://gerrit.wikimedia.org/r/545942

Change 545870 merged by jenkins-bot:
[mediawiki/extensions/Cite@master] Stop sampling when Reference Previews is enabled

https://gerrit.wikimedia.org/r/545870

Change 545869 abandoned by Awight:
Report for ReferencePreviews popups

Reason:
merge with parent.

https://gerrit.wikimedia.org/r/545869

Change 542419 had a related patch set uploaded (by Awight; owner: Awight):
[analytics/reportupdater-queries@master] [WIP] New reports for Reference Previews

https://gerrit.wikimedia.org/r/542419

I think the reportupdater queries are ready, but still waiting for our new schemas to roll out so I can smoke test the ReferencePreviewsCite schema query.

awight added a subscriber: elukey.Tue, Oct 29, 3:00 PM

as far as I know we don't use the graphite writer.

Uh, oh! I think I'm pretty far out on a limb in that case. /me tosses a pinch of salt over their left shoulder

Change 542419 merged by Mforns:
[analytics/reportupdater-queries@master] New reports for Reference Previews

https://gerrit.wikimedia.org/r/542419

Need to tweak the ReferencePreviewsCite report...

Change 549054 had a related patch set uploaded (by Awight; owner: Awight):
[analytics/reportupdater-queries@master] Fix nonexistent field in query

https://gerrit.wikimedia.org/r/549054

awight changed the point value for this task from 8 to 3.Wed, Nov 6, 1:13 PM

Change 549054 merged by Mforns:
[analytics/reportupdater-queries@master] Fix nonexistent field in query

https://gerrit.wikimedia.org/r/549054

There are many more Popups pageviews than I had expected (or many fewer from Cite). Look into this, and tweak the tracking to be idempotent in case that's the issue.

Change 549820 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Popups@master] Ensure that pageviews are recorded at most once

https://gerrit.wikimedia.org/r/549820

Change 549820 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] Ensure that pageviews are recorded at most once

https://gerrit.wikimedia.org/r/549820

Change 550104 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Popups@wmf/1.35.0-wmf.5] Ensure that pageviews are recorded at most once

https://gerrit.wikimedia.org/r/550104

Change 550104 abandoned by Awight:
Ensure that pageviews are recorded at most once

Reason:
This doesn't solve our problem.

https://gerrit.wikimedia.org/r/550104

I'm failing to find a simple answer to the pageviews thing. I did discover that the Popups pageviews include Special page and Edit page impressions, but these alone shouldn't add up to the 2.5x disparity found in the real data. I'm going to wrap up this phase, and put a warning on the graph with suspect data. If anything, the difference will *under*-estimate the impact of Reference Previews, and we're already getting much higher popup view rates than baseline footnote views.

One quick fix would be to recalculate the ratios using Cite pageview counts, but this doesn't feel safe either.

Change 550489 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Popups@master] [WIP] Reduce double-counting of Reference Previews pageviews

https://gerrit.wikimedia.org/r/550489

Change 550489 abandoned by Awight:
[WIP] Reduce double-counting of Reference Previews pageviews

https://gerrit.wikimedia.org/r/550489

Change 550674 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Popups@master] [WIP] Don't record Popups actions on non-content pages

https://gerrit.wikimedia.org/r/550674

Change 550675 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/Cite@master] Track pageviews only on content page views, not edits

https://gerrit.wikimedia.org/r/550675

Change 550683 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/Popups@master] [WIP]

https://gerrit.wikimedia.org/r/550683

Change 550675 merged by jenkins-bot:
[mediawiki/extensions/Cite@master] Track pageviews only on content page views, not edits

https://gerrit.wikimedia.org/r/550675