Referrer policy for browsers which only support the old spec
Open, NormalPublic

Description

Edge and Safari support an older version of the Referrer Policy spec, which only recognizes never, always, origin and default (caniuse). In T87276 we have set origin-when-cross-origin; that causes Edge and Safari to use the default policy (full URL when target is HTTPS; empty string when target is HTTP) which is both too restrictive and not restrictive enough.

We should use the fallback mechanism defined in the spec and define something more appropriate for those browsers. That would mean:

  • allow $wgReferrerPolicy to be an array, and output multiple <meta> tags in that case;
  • set our referrer policy to something like [ 'origin-when-cross-origin', 'origin' ] (that would prevent Edge/Safari from sending the full URL for same-domain requests but otherwise be the same; it's possible they are not sending it already, per T148780#2891117), or maybe [ 'origin-when-cross-origin', 'origin-when-crossorigin', 'origin' ] (if there is still some browser out there which supports the somewhat-less-old version of the spec which did have "Origin When Cross-Origin" but had a typo in the keyword).
Tgr created this task.Nov 19 2017, 10:05 PM
Restricted Application added a project: Operations. · View Herald TranscriptNov 19 2017, 10:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Tgr updated the task description. (Show Details)Nov 19 2017, 10:06 PM
Ottomata moved this task from Incoming to Radar on the Analytics board.Nov 20 2017, 4:40 PM
BBlack added a subscriber: BBlack.Dec 22 2017, 1:47 PM

Open Q's from my POV:

  1. Flipping these Edges/Safaris to origin is going to deny us information on internal referrers that we currently get from all browsers. Do we care and use/analyze this data?
  2. Have we confirmed the actual behavior of UAs in question? As in, actually fired up the most-recent and other less-recent-but-still-popular versions of these browsers and confirmed their capabilities and behaviors wrt to referrer policy?
  3. In the past, I'm pretty sure there were some popular browsers (maybe Safari) that did in fact need the typo'd variant origin-when-crossorigin. I don't think we considered the fallback mechanism for this case.
  4. For the cases where, given everything above, fallback using some alternate policy string is preferable to what we're doing today, do the browsers in question change behavior as expected when given a fallback policy (or do they also fail to implement fallback properly?).
BBlack moved this task from Triage to Watching on the Traffic board.Dec 22 2017, 1:48 PM

Change 399998 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/core@master] Allow multiple values for $wgReferrerPolicy

https://gerrit.wikimedia.org/r/399998

Tgr added a subscriber: Nuria.Dec 23 2017, 6:11 PM

Tested in current Chrome and it behaves as expected.

@Nuria: who would be the best person to answer BBlack's first question?

Change 399998 merged by jenkins-bot:
[mediawiki/core@master] Allow fallbacks for $wgReferrerPolicy

https://gerrit.wikimedia.org/r/399998

Nuria added a comment.Jan 3 2018, 12:53 AM

Flipping these Edges/Safaris to origin is going to deny us information on internal referrers that we currently get from all browsers.

Let's see. Now, for browsers that support our policy, we get:

  • origin if external site
  • full url if internal

For Safari we just get incomplete data. I think the premise that Safari is sending us "full" urls by default by virtue of its default policy might be wrong. It has to be wrong if Safari sessions are shorter according to this ticket:
https://phabricator.wikimedia.org/T148780

So, if anything I think this change will make that data appear. The issue is also not restricted to Safari but to other older browsers

FYI, Safari does support "origin-when-cross-origin" now https://bugs.webkit.org/show_bug.cgi?id=154588 but i assume if it finds both tags it will use the better one (well... not sure if this fix has even been released)

In the past, I'm pretty sure there were some popular browsers (maybe Safari) that did in fact need the typo'd variant origin-when-crossorigin. I don't think we considered the fallback mechanism for this case.

True but I think that has now been corrected for browsers that support the policy, that was discussed on original ticket: https://phabricator.wikimedia.org/T87276

Ottomata triaged this task as Normal priority.Jan 16 2018, 8:09 PM
Tgr added a comment.Feb 2 2018, 4:06 AM

@Nuria any thoughts about the next step? Should we just enable the fallback and check the data to see if it had any unexpected effect? Or do we need to do some kind of testing / experiment first?

Nuria added a comment.EditedFeb 2 2018, 3:40 PM

I see no problem with changset, my comment was about pointing out that "Flipping these Edges/Safaris to origin is going to deny us information on internal referrers that we currently get from all browsers." is probably incorrect,

Nuria added a comment.Feb 2 2018, 3:47 PM

Safari sessions still appear "shorter":

Compare to chrome ones:

So Safari is sending us no data for internal referrals to this day.

Tgr added a comment.Feb 2 2018, 8:41 PM

@Nuria the changeset did not change Wikimedia referrer policy, just made MediaWiki able to do so. What I would like is to extend the current origin-when-cross-origin policy with the origin-when-crossorigin, origin fallback chain. (Also maybe we could pick a few smaller wikis for different combinations, just to see if origin-when-crossorigin does any real work?) Do you think that needs more investigation into browser behavior, or can it go ahead and the investigation can happen by using the collected data?

Nuria added a comment.Feb 3 2018, 12:04 AM

What I would like is to extend the current origin-when-cross-origin policy with the origin-when-crossorigin, origin fallback chain

Right right, my point is that it sounds great, it can only help the current issue with safari.

Do you think that needs more investigation into browser behavior, or can it go ahead and the investigation can happen by using the collected data?

I think we should go ahead.

Change 407872 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Support fallback values for referrer policy

https://gerrit.wikimedia.org/r/407872

Change 407872 merged by jenkins-bot:
[operations/mediawiki-config@master] Support fallback values for referrer policy

https://gerrit.wikimedia.org/r/407872

Mentioned in SAL (#wikimedia-operations) [2018-02-07T00:45:18Z] <ladsgroup@tin> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:407872|Support fallback values for referrer policy (T180921)]] (duration: 01m 12s)

Tgr added a comment.Feb 7 2018, 2:01 AM

@Nuria the new config is live now (although it will only take effect gradually due to Varnish caching). Can you check if the browser behavior improved / did not degrade?

My main concern is that same browser fails to follow the spec in that it gives priority to the first known value, not the last one. If that turns out to be the case we'd have do something like <primary> <fallback1> <fallback2> <fallback1> <primary>.

Nuria added a comment.Feb 7 2018, 6:19 PM

Will do, let's let it bake a bit and i shall check.

Nuria added a comment.Feb 13 2018, 7:54 PM

As of me checking today referral groups are mostly unchanged from the graphs i pasted above. That is, safari sessions appear "artificially" shorter

Tgr added a comment.Feb 13 2018, 8:05 PM

If nothing seems more broken than before that's good enough for me :) The main goal was to prevent Edge and Safari from sending the full URL as a referer to third-party websites when the user leaves Wikipedia.

gh87 added a subscriber: gh87.Apr 2 2018, 8:05 PM

I was using the Apple Safari 11.0.3 (13604.5.6). Then I enabled the "Develop" tab in the Menu, then right-clicked to open the context menu, and then clicked "Inspect Element" to open Safari's developer tools.

Then I went from any article to an external https:// website and typed document=referrer, resulting in "" = $. That means "no-referrer" must have been used.

I went from one page to another within Wikipedia internally while I used the developer tools. I found an error message saying below:

Failed to set referrer policy: The value 'origin-when-crossorigin' is not one of 'no-referrer', 'origin', 'no-referrer-when-downgrade', or 'unsafe-url'. Defaulting to 'no-referrer'.

I must say that Apple Safari does a better job than Microsoft Edge.

Tgr added a comment.Apr 2 2018, 8:16 PM

Apparently it doesn't implement fallbacks as per spec (the last fallback value is origin). I don't think there's anything we can do about that.

Should we open a separate bug for the Safari issue? (See also T87276#4218911)

gh87 added a comment.May 20 2018, 11:32 AM

Should we open a separate bug for the Safari issue? (See also T87276#4218911)

Please feel free, but the task should be fresh, IMO.

TheDJ added a subscriber: TheDJ.EditedMay 20 2018, 12:34 PM

This works now in Safari 11.1 (13605.1.33.1.2) I think:

Wikidata to Wikidata

From wikidata.org to google.

English Wikipedia to Wikidata

It still reports an error, because it doesn't recognize the misspelled value in our chain, but it just skips it now.

gh87 added a comment.May 21 2018, 8:47 PM

From Safari 11.1 (13605.1.33.1.4) at mac OS X High Sierra 10.13.4:

Below happens when going from one article to another:

I also found this error(?) message:

Failed to set referrer policy: The value 'origin-when-crossorigin' is not one of 'no-referrer', 'no-referrer-when-downgrade', 'same-origin', 'origin', 'strict-origin', 'origin-when-cross-origin', 'strict-origin-when-cross-origin' or 'unsafe-url'. Defaulting to 'no-referrer'.
load.php:139:634

Below happens when going from one article to an external website:

TheDJ added a comment.May 23 2018, 7:44 AM

@gh87 yeah, so that is correct. We indicate as referer policy: "origin, origin-when-crossorigin, origin-when-cross-origin". As observed in your screenshots, it sends the full url when you remain on the same domain, and it sends just the domain (origin) when going to a different domain (cross origin). The error is because it doesn't understand the misspelled version (from the draft specification) we have in the fallback chain. Now that error should probably be a warning, but as there is a 3rd policy overwriting origin, and the misspelled version that is set last, the actual effect is correct.

Nuria added a comment.May 23 2018, 4:55 PM

Latest data about this show internal referrers for safari creeping up from Late march:

TheDJ added a comment.May 24 2018, 3:41 PM

Shall we consider this closed then ?

Tgr added a comment.May 24 2018, 3:55 PM

Would be nice if someone could test it on IE or Edge, now that Safari is fixed those are the only two browsers that need a fallback.

TheDJ added a comment.EditedMay 24 2018, 4:02 PM

P.S. I do think that fallback is indeed broken on Safari. Note how the error message says it is reverting the policy to no-referrer, instead of origin. This might be problematic if at a later time we want to switch to potentially even newer policies, not yet supported by Safari 11.1. I will file a webkit bug about it.

RP88 added a subscriber: RP88.Tue, Jul 31, 2:30 PM