Fix broken referer categorization for visits from Safari browsers
Open, NormalPublic

Description

Due to a change we made to our meta referrer tag in February of 2016 (T87276) and a bug in safari, almost all of the internally referred traffic on Safari is miscategorized. Since Safari is our second biggest browser family at ~20% of traffic, this impacts our non-internal referral count which is an essential measure of product health as it is a rough proxy for visits. The ratio is also an important proxy for session depth. This has been broken for almost a year and we should fix it or put more pressure on the webkit folks to fix it on their end: https://bugs.webkit.org/show_bug.cgi?id=154588

I do not know how to fix it, but the original change is heavily discussed here T87276 and the implications for our metrics are outlined here: T148780

JKatzWMF created this task.Jan 5 2017, 9:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 5 2017, 9:17 PM
ema moved this task from Triage to Watching on the Traffic board.Jan 20 2017, 7:07 PM
Ottomata triaged this task as Normal priority.Mar 6 2017, 7:29 PM
Ottomata added a subscriber: Ottomata.

I don't fully understand this issue, but I'm not sure how it could be fixed on our side. If there is a way, its likely to be very complicated. I'm doing some ops triaging this week, and I'm not sure what to do with this ticket.

@Ottomata Thanks for voicing your uncertainty. I am also uncertain/confused about the cause (T87276), or the solution as it is a bit out of my technical depth, but it seems that there is a typo in webkit and I don't understand why we can't adjust for that typo.

Nuria added a comment.Mar 7 2017, 5:09 AM

On our end we have no preference how to fix issue (adding typo back would break other things, but I do not dispute that it might be fine), traffic & reading (and likely performance) just need to agree how to move forward, analytics will work automagically, so there is nothing on the analytics side to fix, the issue is on the mediawiki headers which are (i guess) collectivelly owned by all teams. I think the shortest path to action would be for a mediawiki reading developer to drive issue.

Nuria added a subscriber: mforns.EditedMar 7 2017, 6:07 AM

I was going to summarize but @mforns already did it in a prior ticket. For what is causing this issue see: https://phabricator.wikimedia.org/T148780#2891117

The workarrounds to fix it could be several:

  1. changing to the typo'd version of meta tag, that be changing origin-when-cross-origin,to origin-when-crossorigin on this file: https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L12059 This would mean that Safari sends the right referrer but other browsers might not recognize the typo-ed version (browsers moved to version w/o typo a while back with the notable exception of safari) so we might need both versions of the meta tag to be available.
  1. do not use meta tags and use headers in VCL to carry this information using the new Referrer policy: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy

Both 1 and 2 need browser testing to assess level of support, probably 1) is less work as of now if we can get away with typo-ed version, best choice however is to do 2) as it allows us to modify that header on VCL when we need to even if html is cached. Testing for either option would need to happen in all popular browsers: https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-browser

TheDJ added a comment.Mar 7 2017, 2:06 PM

As far as i'm aware, this is NOT due to a typo. Safari simply implements a very limited set of referrer policies: 'no-referrer', 'origin', 'no-referrer-when-downgrade', or 'unsafe-url'.

I also wonder about IE. As far as I know it also doesn't support this specific policy, but perhaps it falls back to a more 'leaky' referrer policy.

TheDJ added a comment.Mar 7 2017, 2:29 PM

The Referrer-Policy HTTP Header is actually quite different from the meta header. I have created a mock request http://www.mocky.io/v2/58bec319260000201bf07c51, but it doesn't seem to me that Safari does anything at all with it. I get the full referer in my request console when I click the link.

Nuria added a comment.Mar 7 2017, 4:27 PM

As far as i'm aware, this is NOT due to a typo. Safari simply implements a very limited set of referrer policies: 'no-referrer', 'origin', 'no-referrer-when-downgrade', or 'unsafe-url'.

I see, then solution 1) should work for Safari if we can render a different meta tag for safari, is that aceptable? cc @TheDJ

@JKatzWMF, I think if you really really really want this fixed, you'll need to find a MediaWiki dev to revisit https://meta.wikimedia.org/wiki/Research_talk:Wikimedia_referrer_policy, involve the folks who made this decision in the first place, and then figure out what to do and do it.

Does reading have a developer that could shepherd this?

@Ottomata @Nuria Just spoke with @Nuria and between the above comments and our conversation I think I have what we need to figure out next steps....which are probably conversations with other folks like Dario and the discovery team. I really appreciate the work that went into exploring and communicating this.

Nuria added a comment.Mar 7 2017, 7:02 PM

you'll need to find a MediaWiki dev to revisit https://meta.wikimedia.org/wiki/Research_talk:Wikimedia_referrer_policy

I *think* policy as is works, we will just need to special case safari seems like

Krinkle removed a subscriber: Krinkle.Mar 18 2017, 3:32 AM

+1 to @Nuria 's comment: I think the main concern here from @DarTar and me is that external websites need to have some sort of awareness where the dark traffic is coming from: so if it messes with external analytics tools for referrals to other sites, it is a less-than-good solution.

TheDJ added a comment.Aug 25 2017, 9:13 AM

Safari Technology Preview v38, released 2 days ago, seems to be solving this. I at least am no longer getting the console warning that the policy is unsupported by the browser (have not verified yet, if the behaviour is now actually correct).

This might mean that the next major version of Safari on desktop and mobile, could start showing up in the referrer stats.

Nuria added a comment.Aug 31 2017, 1:35 PM

@TheDJ : noted, please ping us when those changes make it into safari's main version. The bulk of our views is on Safari 10 so if problem is fixed on next version we should see our referrer stats change

TheDJ added a comment.EditedAug 31 2017, 7:46 PM

I have now tested this and the preview does indeed implement origin-when-cross-origin as directed by our meta referrer header.

@TheDJ Thanks for the heads up!

FYI: Seems the change missed the cut-off for the Safari 11 release.

iOS 11 Safari 11 not yet fixed: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1
macOS Safari 11 not yet fixed: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Safari/604.1.38

So possibly no earlier than the March 2018 for this change (Safari 11.1)

Nuria added a comment.Sep 20 2017, 3:02 PM

Super thanks for following up