Page MenuHomePhabricator

Sampling sanity check for Schema:Popups pageview events
Closed, ResolvedPublic

Description

The number of pageLoaded events registered by the Popups schema seems only about 60-70% of what one would expect from the corresponding number of (non-spider) pageviews from Hive, see these three charts from the experiments on ruwiki, itwiki and huwiki. It seems worth determining whether this can be explained by natural factors (two possibilities listed below) or points to an issue in the instrumentation.

(Restricted to Firefox-like user agents to circumvent the general EventLogging data quality issues from T146840 .)
The y-axis maximum is selected as the sampling rate on each of these projects, per https://meta.wikimedia.org/wiki/Schema_talk:Popups . NB: the schema samples per browser session, not per pageview, so part of the discrepancy will come from session cookies not being set or preserved. If this is a factor, one would expect the discrepancy a bit smaller when the sampling rate is higher.

As @dr0ptp4kt pointed out earlier, DNT is another factor, but probably not large enough to explain a 30-40% drop.

Data source

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 21 2016, 4:56 PM

@Tbayer I wonder if you see the same results when looking at events coming from Firefox and Chrome? Support for the Beacon API is lousy [1] which maybe causing this data mismatch.

[1] http://caniuse.com/#feat=beacon

@bmansurov The data above is already restricted to Firefox, and the consistency of sendBeacon support was indeed another reason to do so (besides the EL issues from T146840 which I mentioned in the task description).

ovasileva triaged this task as High priority.Oct 26 2016, 10:59 AM
ovasileva changed the task status from Open to Stalled.Nov 2 2016, 7:43 PM
ovasileva lowered the priority of this task from High to Normal.
ovasileva moved this task from Incoming to Triaged but Future on the Readers-Web-Backlog board.
Jdlrobson assigned this task to phuedx.Apr 17 2017, 11:25 PM
Jdlrobson moved this task from Triaged but Future to Upcoming on the Readers-Web-Backlog board.
Jdlrobson added subscribers: phuedx, Jdlrobson.

@Tbayer and @phuedx have we sanity checked or should we do this next sprint?

phuedx reassigned this task from phuedx to Tbayer.Apr 20 2017, 5:12 PM

...

@Tbayer: Am I correct in assuming that we'll re-run this sanity check after T161769: Schema:Popups sends extraneous link interaction events in control condition is resolved?

Yes, I have been intending to update these charts after the extraneous events issue is resolved. Per the recent results there regarding browser families and our discussion with @ovasileva earlier today, I may now also consider to do that earlier while restricting it to Chrome, although (considering that we restricted to Firefox earlier) that might introduce additional complexities - doing it after the fixes would still be preferable.

Note that the task was not about me (re)generating this data, but about the team giving an assessment whether this discrepancy (if it continues to be present now after the rewrite) can be explained by natural factors (e.g. the two possibilities already listed in the task description) or points to an issue in the instrumentation.

@Tbayer and I will be able to discuss this during Tuesday's "Data quality sync" meeting.

phuedx claimed this task.May 3 2017, 5:54 AM
phuedx moved this task from Needs Analysis to Doing on the Reading-Web-Sprint-96 board.
phuedx added a comment.EditedMay 3 2017, 9:50 PM

As you've noted in the description, we sample based on a session cookie not per pageview. If the session cookie weren't set, then I'd expect the number of events to tend towards the expected number.

As @dr0ptp4kt noted, DNT is definitely a factor.

I'd say that requiring that we only log events via sendBeacon, support for the Beacon API is a major contributing factor.

phuedx reassigned this task from phuedx to Tbayer.May 4 2017, 9:39 AM
phuedx removed a project: Reading-Web-Sprint-96.

Removing this from the current sprint for now as it's blocked on T163198: Track instances of duplicate Popups events being logged.

Jdlrobson changed the task status from Stalled to Open.May 21 2017, 11:22 AM

T163198 is resolved.

Two notes:

Tbayer closed this task as Resolved.Sep 1 2018, 7:13 AM

The Popups instrumentation and the page previews feature in general subsequently went through various rewrites and bug fixes, making this task obsolete.

In retrospect, at least part of the discrepancy here was caused by T175918: EventLogging subscriber module in ready state but not sending tracked events (which was discovered by other means - also meaning that we could have found that bug 11 months earlier by investigating more thoroughly in the context of this task..)