Page MenuHomePhabricator

Instrument print to PDF button
Closed, ResolvedPublic3 Estimated Story Points

Assigned To
Authored By
ovasileva
Nov 24 2017, 12:35 PM
Referenced Files
F13956453: Screen Shot 2018-02-22 at 1.37.54 PM.png
Feb 22 2018, 9:39 PM
F13956456: Screen Shot 2018-02-22 at 1.37.43 PM.png
Feb 22 2018, 9:39 PM
F13812767: Screen Shot 2018-02-16 at 5.53.29 PM.png
Feb 17 2018, 1:58 AM
F12752870: image.png
Jan 23 2018, 7:51 PM
F12752874: image.png
Jan 23 2018, 7:51 PM
F12752876: image.png
Jan 23 2018, 7:51 PM
F12752865: image.png
Jan 23 2018, 7:51 PM
F12752879: image.png
Jan 23 2018, 7:51 PM
Tokens
"Burninate" token, awarded by Jdlrobson.

Description

Background

Based on several bug reports, we would like to know more about which click impressions result in a beforeprint event. The hope is this might shed more light on the issue about which browser versions of Chrome are having issues with printing the PDF button (i.e. on which browsers does a user select the button but nothing happens, see T181261: Provide system feedback while the Android opens up the print dialog using download article action).

Currently we log an event action clickPrintableVersion when the "Printable version" link is clicked in desktop. We want to add the mobile equivalent.

Acceptance criteria

Add the following events to the print instrumentation schema for Minerva only:

  • When the "download" button is clicked in the Minerva skin a "clickPrintableVersion" event will be sent. No change to schema is needed for this event.
  • When the "download" button is visible in the Minerva skin a "shownPrintButton" event will be sent to signal a page impression. A schema change for this new event is needed.
  • File a task for the Analytics Engineering team to blacklist this schema from MariaDB/MySQL (like we did for Popups), due to the high event rate expected to be caused by the button impressions event
  • Create a task for turning schema off after n weeks of collected data

--> T196159

Notes

Note, per analysis by @pmiazga we cannot reliably detect:

  • Print dialog opened
  • how long does it take for dialog to appear after button is pressed

Developer notes

It's probably easiest to implement this by augmenting the existing (and already partially vetted) Print schema, e.g. by extending the definition of the clickPrintableVersion action to Minerva, standing for clicks on the PDF button there.

Triggering the shownPrintButton event will be tricky as the schema is managed by WikimediaEvents but added by the Minerva skin. I propose the Minerva skin triggers an event e.g. mw.track and WikimediaEvents makes use of trackSubscribe to forward the event in EventLogging.

Testing criteria

  • Bucket yourself by running the following code in your developer console on the BETA CLUSTER:
/*!
 * Track browser print events
 *
 * Each action is only logged once per page view. That is, no matter how many
 * times the user clicks on 'Printable Version', only one
 * 'clickPrintableVersion' action is logged for that page until the page is
 * refreshed. Ditto the `onBeforePrint` action.
 *
 * @see https://phabricator.wikimedia.org/T169730
 * @see https://meta.wikimedia.org/wiki/Schema:Print
 */
( function ( $, track, trackSubscribe, config, user, mwExperiments ) {


	/**
	 * Whether the user session is sampled
	 *
	 * @param {number} samplingRate - a float between 0 and 1 for which events
	 *   in the schema should be logged.
	 * @return {boolean}
	 */
	function isInSample( samplingRate ) {
		var bucket = mwExperiments.getBucket( {
			name: 'WMEPrint',
			enabled: true,
			buckets: {
				control: 1 - samplingRate,
				A: samplingRate
			}
		}, user.sessionId() );
		return bucket === 'A';
	}

  while ( !isInSample( 0.1 ) ) {
    mw.storage.session.remove('mwuser-sessionId' )
  }
}( jQuery, mediaWiki.track, mediaWiki.trackSubscribe, mediaWiki.config, mediaWiki.user, mediaWiki.experiments ) );
  • Verify that when clicking the download icon an event is fired.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ovasileva changed the point value for this task from 5 to 3.Dec 13 2017, 6:21 PM

@ovasileva I had a read through the discussion here and attempted to summarize ^
It looks good to go to me, but I'm a little confused at what was decided in the conversation between @Tbayer and @Nirzar . If you could edit the description with that conclusion, I'd appreciate it!

I don't think that aspect had been decided already. We discussed this in kickoff today and decided to add the instrumentation for the button impressions (see below).

we have added 3 conditions to show the button. chrome/android + ability to print + namespace. and I'm just not sure adding those three conditions to look up pageviews will give us the correct number. the logic for conditions may vary.

Recap: If the three conditions were only based on browser type, OS version and namespace, it should have been possible to match them in the standard pageview data. But it looks like the actual method in the code to decide whether to show the button may be more complex, and becoming more complex still (per T182059 etc.).

is there a large downside of getting impressions of the button?

Summarizing what I said in kickoff: We can do it, it will just increase the number of events by orders of magnitude, so we needed a rationale for doing so, and need to ask Analytics Engineering to blacklist the schema from MySQL (because it is now likely to exceed the recommended event rate for MySQL). Since we already went through this with the Popups schema (which includes a similar high-volume event, "pageLoaded"), I'm not too worried about that aspect.

Change 403181 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[mediawiki/skins/MinervaNeue@master] Send events when user taps download icon

https://gerrit.wikimedia.org/r/403181

@ovasileva @Tbayer do we also want to track:

  • ignore button clicks - happens when we user spams download button but the system is still processing the previous click ( eg it's still downloading images)
  • the time it took from button click to window.print() system call (time system spend on pre-loading images, max 3s)

@ovasileva @Tbayer do we also want to track:

  • ignore button clicks - happens when we user spams download button but the system is still processing the previous click ( eg it's still downloading images)

Currently, this shouldn't be possible right? The spinner appears as soon as the button is clicked while we're loading the images

  • the time it took from button click to window.print() system call (time system spend on pre-loading images, max 3s)

this would only be a measure of how long it takes us to load all the images, correct? If so, I don't think it's necessary to track.

Currently, this shouldn't be possible right? The spinner appears as soon as the button is clicked while we're loading the images

Yes, but it's still a button which you can click, we change the icon from download to spinner and ignore clicks. Even this click does nothing we can still track it.

this would only be a measure of how long it takes us to load all the images, correct? If so, I don't think it's necessary to track.

Correct, that would be the time it takes to load the images. This value can tell us how much time users have to wait before we call window.print(). Currently, this time is limited to 3 seconds (we wait for all images to load, or 3 seconds whatever happens first). There is a possible scenario when many users have to wait ~3seconds - that would mean that we should bump the timeout as we're not able to fetch all images for user - printed version might be missing some images.

Change 403214 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[mediawiki/extensions/WikimediaEvents@master] Listen to minerva download events and log events accordingly

https://gerrit.wikimedia.org/r/403214

Change 403181 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Send events when user taps download icon

https://gerrit.wikimedia.org/r/403181

Change 403214 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Listen to minerva download events and log events accordingly

https://gerrit.wikimedia.org/r/403214

Sorry I should have caught this during code review but there are two issues:

Change 403765 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[mediawiki/extensions/WikimediaEvents@master] Fix incorrectr mediawiki.trackSubscribe() call

https://gerrit.wikimedia.org/r/403765

Change 403765 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Fix incorrect mediawiki.trackSubscribe() call

https://gerrit.wikimedia.org/r/403765

Over to you Anthony! Hopefully the instructions are clear, let me know if you need any help! Note: I had some caching issues when I myself tried to verify this but I can confirm I've seen this working!

Sorry for the delay. I got a little sidetracked by the bucketing steps above. I have not actually done those, but I have done a fair amount of testing the download button on various android/chrome devices. So far everything seems to be working as expected. As expected, on older chrome, non-chrome, and iOS I am not seeing the download button. The download button seems to be working on all the newer chrome browsers I have tested.

Oh, I may have spoken too soon or perhaps I am not in the right bucket. I am now noticing some issues with non-chrome browsers. This is all on android: It looks like Chrome and Firefox are working. I am seeing the download icon on Opera Mobile, Opera Mini, Maxthon and Dolphin.

Target URL: https://en.wikipedia.beta.wmflabs.org/wiki/Math

ChromeFirefoxUC BrowserOpera MobOpera MinMaxthonDolphin
CorrectCorrectCorrectIncorrectIncorrectIncorrectIncorrect
image.png (1×720 px, 187 KB)
image.png (1×720 px, 183 KB)
image.png (1×720 px, 196 KB)
image.png (1×720 px, 155 KB)
image.png (1×720 px, 168 KB)
image.png (1×720 px, 172 KB)
image.png (1×720 px, 478 KB)

Note - in all the above incorrect cases, tapping download doesn't appear to do anything.

Jdlrobson assigned this task to Jdrewniak.
Jdlrobson added a subscriber: ABorbaWMF.

Jan in standup said he'd take a look at this problem and see what's going on.

I can confirm that the download button is visible on the browsers mentioned above (Opera Mini/mobile, Maxthon, Dolphin etc).
The reason for this (at least for Opera) is because opera is based on Chromium, so when we check for Chrome, it returns true. This wouldn't be a problem it acted like Chrome and showed the native print dialog, but unfortunately it don't, and therefore... the print button doesn't work on some chrome based browsers 😭.

Dolphin and Maxthon are based on Webkit, so I'll investigate why they're testing true for the chrome test.

I'll try to debug this further to see if there's anything we can do, but I won't have a real android device until next week.

Issue is caused by T182059 so let's not confuse things on this task which is about instrumentation...
@ABorbaWMF can you confirm the EventLogging is working currently and can we move this to sign off?

Hi all!

We've noticed that the Print schema has been sending events to EL for a couple days.
But mysql is receiving a lot of events, around 200 per second.
Is that expected?

This volume is over what mysql can stand, after a couple weeks the table will be so huge that it will be very difficult to query/maintain.
In EL's documentation we indicate that a nice maximum for mysql storage is 10 events per second.
We are going to blacklist it for mysql insertion, sorry.

The data will still be normally inserted, available and queryable in Hive's EventLogging refined tables.

Niedzielski subscribed.

Thanks @mforns and sorry for the trouble. We had intended to notify Analytics of this high volume metric before it went out :|

Hi @Niedzielski , no problem.

Would you like to reduce the sampling rate?
If so, we can leave the schema in mysql.

Otherwise, yea, you'll have the data in Hive.

Please think about reducing the sampling rate and let us know!
We'll wait until tomorrow to blacklist the schema.

Cheers!

@mforns, as @phuedx pointed out to me moments ago, we actually intended to blacklist the schema per the task description:

File a task for the Analytics Engineering team to blacklist this schema from MariaDB/MySQL (like we did for Popups), due to the high event rate expected to be caused by the button impressions event

We may wish to reduce the rate in the future but for now the high throughput is wanted. Please proceed with blacklisting the schema!

I 'believe' this is working now thanks to help from @Jdlrobson. I'm not sure the extension was working, probably a pilot error, but I am seeing events.

Screen Shot 2018-02-16 at 5.53.29 PM.png (1×2 px, 1 MB)

Great, thanks! Were you also able to check that the events contained the correct value for each field as described in the schema? (In particular, the session token should be the same for each of these three events, and generally for all events sent during that browser session.)

Ok, once more thanks to @Jdlrobson I was able to take a look. I think it is ready for sign off:

Screen Shot 2018-02-22 at 1.37.43 PM.png (1×2 px, 1 MB)

Screen Shot 2018-02-22 at 1.37.54 PM.png (1×2 px, 1 MB)

looks good, thanks @ABorbaWMF - I think it makes most sense for @Tbayer to sign this off

The request volume of this schema is quite high: https://grafana.wikimedia.org/dashboard/db/eventlogging?orgId=1&from=now-7d&to=now-5m about 200 requests per second.
Seems like the questions posed by this ticket can probably be already answered by the data we have, is anyone planning on whether we can turn this schema off? 200 requests per sec for a week is 120 million of data points.

@Nuria I imagine your comment refers to Hive data, because it's big enough in that context too.
But just for the record, this schema is blacklisted from mysql since 20180211173030.

Right, it is not a disk space concern but rather, my concern is that we have now millions of data points that nobody is looking at, it is happen before that there were issues with instrumentation/sampling. Seems like we should take time to peek at data and see whether it serves the purpose we want it for, if yes maybe collecting can be stopped. If not let's correct before we have poor data of no use.

@Nuria - this task is to ensure the instrumentation was implemented correctly. Our plans for analysis of the data are tracked under T179915: Determine expected amount of usage of mobile print to PDF button per browser. After that task is done, we might have some follow-up/general questions, but we would most likely be ready to pause data collection.

@ovasileva until someone takes a little bit of a time to vet data I do not think you can assume instrumentation was implemented correctly, "data flows in" sure. Now, there should be a QA process after data collection to make sure the data is indeed suited for analysis so we do not find ourselves in the situation of collecting tons of data we later need to throw away (this is happened just recently).

This task has a somewhat a convoluted history, but after spending some time reviewing the various parts I think we can check all the boxes and "sign off". The only part that is not entirely clear to me from the comments above is whether anyone tested the standard print path ("Share" --> "Print") for Chrome mobile (per T179915). Clarifying that might still be useful, but it doesn't block the main analysis at T179915 regarding the download button usage.

I did some vetting of the data about a month ago (when we made first use of it for our quarterly check-in), running various sanity checks whose results I'm copying below for reference. E.g. the ratio of shown print buttons to pages viewed on Chrome mobile was roughly 0.69 on May 1 (keeping in mind the 10% sampling rate), which is a bit low but not entirely implausible (taking into account DNT etc.)
No red flags so far, so I'm closing this task.

SELECT event.action, COUNT(*) FROM event.print
WHERE year = 2018 AND month = 5 AND day = 1 
GROUP BY event.action;

onBeforePrint	41014
shownPrintButton	11485449
clickPrintableVersion	17787
3 rows selected (22.153 seconds)


SELECT event.skin, COUNT(*) AS events FROM event.print
WHERE year = 2018 AND month = 5 AND day = 1 
GROUP BY event.skin;

event.skin	events
other	5
minerva	11517464
vector	26781
3 rows selected (22.3 seconds)


SELECT event.action, COUNT(*) AS events 
FROM event.print
WHERE year = 2018 AND month = 5 AND day = 1
AND event.skin = 'minerva'
GROUP BY event.action;
event.action	events
onBeforePrint	15460
shownPrintButton	11485449
clickPrintableVersion	16555
3 rows selected (30.4 seconds)



SELECT useragent.browser_family AS browser, COUNT(*) AS buttons_shown
FROM event.print
WHERE year = 2018 AND month = 5 AND day = 1
AND event.skin = 'minerva'
AND event.action = 'shownPrintButton'
GROUP BY useragent.browser_family
ORDER BY buttons_shown DESC LIMIT 100;

browser	buttons_shown
Chrome Mobile	9584642
Samsung Internet	1401545
Chrome	374445
Amazon Silk	50996
Edge Mobile	40521
Yandex Browser	24957
Edge	3677
Puffin	3538
Opera	501
[...]
28 rows selected (48.261 seconds)



SELECT SUM(view_count) AS pageviews
FROM wmf.pageview_hourly
WHERE year = 2018 AND month = 5 AND day = 1
AND user_agent_map['browser_family'] = 'Chrome Mobile';

INFO  : OK
pageviews
138236276
1 row selected (44.228 seconds)



SELECT event.skin AS skin, COUNT(*) AS buttons_shown
FROM event.print
WHERE year = 2018 AND month = 5 AND day = 1
AND useragent.browser_family = 'Chrome Mobile'
AND event.action = 'shownPrintButton'
GROUP BY event.skin 
ORDER BY buttons_shown DESC LIMIT 100;

skin	buttons_shown
minerva	9584642
1 row selected (46.247 seconds)


SELECT event.action AS action, COUNT(*) AS events
FROM event.print
WHERE year = 2018 AND month = 5 AND day = 1 
AND useragent.browser_family = 'Chrome Mobile'
AND event.skin = 'minerva'
GROUP BY event.action;

action	events
onBeforePrint	12612
shownPrintButton	9584642
clickPrintableVersion	14287
3 rows selected (26.166 seconds)