@Ottomata Thanks for voicing your uncertainty. I am also uncertain/confused about the cause (T87276), or the solution as it is a bit out of my technical depth, but it seems that there is a typo in webkit and I don't understand why we can't adjust for that typo.

On our end we have no preference how to fix issue (adding typo back would break other things, but I do not dispute that it might be fine), traffic & reading (and likely performance) just need to agree how to move forward, analytics will work automagically, so there is nothing on the analytics side to fix, the issue is on the mediawiki headers which are (i guess) collectivelly owned by all teams. I think the shortest path to action would be for a mediawiki reading developer to drive issue.

I was going to summarize but @mforns already did it in a prior ticket. For what is causing this issue see: https://phabricator.wikimedia.org/T148780#2891117

The workarrounds to fix it could be several:

1. changing to the typo'd version of meta tag, that be changing origin-when-cross-origin,to origin-when-crossorigin on this file: https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L12059 This would mean that Safari sends the right referrer but other browsers might not recognize the typo-ed version (browsers moved to version w/o typo a while back with the notable exception of safari) so we might need both versions of the meta tag to be available.

2. do not use meta tags and use headers in VCL to carry this information using the new Referrer policy: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy

Both 1 and 2 need browser testing to assess level of support, probably 1) is less work as of now if we can get away with typo-ed version, best choice however is to do 2) as it allows us to modify that header on VCL when we need to even if html is cached. Testing for either option would need to happen in all popular browsers: https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-browser

As far as i'm aware, this is NOT due to a typo. Safari simply implements a very limited set of referrer policies: 'no-referrer', 'origin', 'no-referrer-when-downgrade', or 'unsafe-url'.

I also wonder about IE. As far as I know it also doesn't support this specific policy, but perhaps it falls back to a more 'leaky' referrer policy.

The Referrer-Policy HTTP Header is actually quite different from the meta header. I have created a mock request http://www.mocky.io/v2/58bec319260000201bf07c51, but it doesn't seem to me that Safari does anything at all with it. I get the full referer in my request console when I click the link.

As far as i'm aware, this is NOT due to a typo. Safari simply implements a very limited set of referrer policies: 'no-referrer', 'origin', 'no-referrer-when-downgrade', or 'unsafe-url'.

I see, then solution 1) should work for Safari if we can render a different meta tag for safari, is that aceptable? cc @TheDJ

@JKatzWMF, I think if you really really really want this fixed, you'll need to find a MediaWiki dev to revisit https://meta.wikimedia.org/wiki/Research_talk:Wikimedia_referrer_policy, involve the folks who made this decision in the first place, and then figure out what to do and do it.

Does reading have a developer that could shepherd this?

@Ottomata @Nuria Just spoke with @Nuria and between the above comments and our conversation I think I have what we need to figure out next steps....which are probably conversations with other folks like Dario and the discovery team. I really appreciate the work that went into exploring and communicating this.

Great! :)

you'll need to find a MediaWiki dev to revisit https://meta.wikimedia.org/wiki/Research_talk:Wikimedia_referrer_policy

I *think* policy as is works, we will just need to special case safari seems like

+1 to @Nuria 's comment: I think the main concern here from @DarTar and me is that external websites need to have some sort of awareness where the dark traffic is coming from: so if it messes with external analytics tools for referrals to other sites, it is a less-than-good solution.

Safari Technology Preview v38, released 2 days ago, seems to be solving this. I at least am no longer getting the console warning that the policy is unsupported by the browser (have not verified yet, if the behaviour is now actually correct).

This might mean that the next major version of Safari on desktop and mobile, could start showing up in the referrer stats.

@TheDJ : noted, please ping us when those changes make it into safari's main version. The bulk of our views is on Safari 10 so if problem is fixed on next version we should see our referrer stats change

I have now tested this and the preview does indeed implement origin-when-cross-origin as directed by our meta referrer header.

@TheDJ Thanks for the heads up!

FYI: Seems the change missed the cut-off for the Safari 11 release.

iOS 11 Safari 11 not yet fixed: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1
macOS Safari 11 not yet fixed: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Safari/604.1.38

So possibly no earlier than the March 2018 for this change (Safari 11.1)

Super thanks for following up

@Nuria so this should work now, can we confirm that from the stats ?

@TheDJ is there a way to know when the fix landed in the safari version users have (not when it was merged and such). I just want to .. ahem... triple check that bump we see around March 29th.

@Nuria Safari 11.1 was released with iOS 11.3 and macOS 11.3.4 (as well as macOS 10.12.6 and 10.11.6)

Both realeased to the general public on March 29th
https://www.apple.com/newsroom/2018/03/ios-11-3-is-available-today/
http://osxdaily.com/2018/03/29/macos-high-sierra-10-13-4-update-mac/

If we change the scale on that graph im sure we can find public beta and developer traffic starting early february.

PS this make me wonder how webkit technology preview is counted in our user agent stats......

@TheDJ Thanks for monitoring this and letting us know! It's a huge relief.

So nice when things add up! I can check on the preview staff, let me know if you want that done

I added an entry about this to the log at https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Pageview_hourly#Changes_and_known_problems_since_2015-06-16