Pages with single quote in title are inaccessible by some clients (redirect loop)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Seb35
	Jul 24 2015, 8:38 AM

Description

Web browsers

It seems all current browsers (Firefox 39, Chromium 31 and Opera 30) have no issue with our url encoded single quote ('), but Opera 12 falls into a redirect loop and finally displays a blank page. I know Opera 12.16 is a browser which is becoming old, I’m accustomed to.

This is similar to T105265 and probably comes from 155d555b83eca6403e..

In the Opera Developer Tools (Dragonfly):

Opera fetches URL: https://fr.wikipedia.org/wiki/O'Hare_Branch
MediaWiki server redirects to: https://fr.wikipedia.org/wiki/O%27Hare_Branch
Opera enforces the url to have an unencoded single quote and before fetching from network rewrites the url as: https://fr.wikipedia.org/wiki/O'Hare_Branch
Back to #1.

For now I have not searched if the single quote has the same status in RFC 3986 than the tilde of T105265.

Search engines

(merged from T112425)

@eranroz wrote

New pages with apostrophe characters aren't indexed by external web search engines.

Exmples
English Wikipedia examples:

https://en.wikipedia.org/wiki/Conference_USA_Men%27s_Soccer_Freshman_of_the_Year (works in Bing, doesn't work in Google and Baidu)

https://en.wikipedia.org/wiki/Holland%27s_Next_Top_Model_(cycle_8) (works in Bing, doesn't work in Google and Baidu)

Hebrew Wikipedia examples:

https://he.wikipedia.org/wiki/%D7%9E%D7%99%D7%A7%D7%94_%D7%91%D7%96'%D7%96'%D7%99%D7%A0%D7%A1%D7%A7%D7%99 (created at 30 August 2015; doesn't yield results in Google and Bing)

https://he.wikipedia.org/wiki/%D7%92'%D7%A8%D7%9E%D7%99_%D7%A7%D7%95%D7%A8%D7%91%D7%99%D7%9F (created at 14 August 2015; doesn't yield results in Bing and Google)

See also

https://duck.co/forum/thread/15098/hebrew-wikipedia-articles-with-in-their-names

https://productforums.google.com/d/msg/webmasters/wTPr_r_73sc/KyYsScvOnB4J

Google Translate

(merged from T122786)

@Schnark wrote:

Trying to translate https://fr.wikipedia.org/wiki/O'Hare_Branch via https://translate.google.com/translate?hl=en&sl=fr&tl=en&u=https%3A%2F%2Ffr.wikipedia.org%2Fwiki%2FO%27Hare_Branch&sandbox=1 just gives a complain about an infinite loop.

Web servers

Details

Subject	Repo	Branch	Lines +/-
Revert "MediaWiki.php: Redirect non-standard title urls to canonical"	mediawiki/core	REL1_27	+11 -21
Revert "MediaWiki.php: Redirect non-standard title urls to canonical"	mediawiki/core	master	+11 -21
Revert "MediaWiki.php: Redirect non-standard title urls to canonical"	mediawiki/core	REL1_28	+11 -21
Revert "Do not encode "'" as %27 (redirect loop in Opera 12)"	mediawiki/core	wmf/1.26wmf22	+30 -31
Revert "Do not encode "'" as %27 (redirect loop in Opera 12)"	mediawiki/core	master	+30 -31
Do not encode "'" as %27 (redirect loop in Opera 12)	mediawiki/core	master	+31 -30

Customize query in gerrit

Related Objects

Mentioned In: T138093: Investigate query parameter normalization for MW/services
T276173: Unable to share link to article via Apple Message if title ends with exclamation mark
T257966: URL shall be terminated by %2E or _ if page name ends with dot
T120085: RFC: Serve Main Page of Wikimedia wikis from a consistent URL
T144100: Pageview dumps incorrectly formatted, need to escape special characters
T67402: URLs for the same title without extra query parameters should have the same canonical link
T122786: Pages with apostrophe character in their title can't be translated by Google Translate
T116986: Set up UrlShortener dumps
T104755: Move URL-routing logic into MediaWiki
T112425: Pages with apostrophe character in their title aren't indexed by external search engines
T111933: Certain edit links on de.wp redirect to Main Page due to "NoScript"
T112069: Instability on fr.wikiversity project
rMW7b3d7154fe4b: Revert "Do not encode "'" as %27 (redirect loop in Opera 12)"
rMW8477b1b277c0: Revert "Do not encode "'" as %27 (redirect loop in Opera 12)"
rMWa89a21990e9d: Do not encode "'" as %27 (redirect loop in Opera 12)
Mentioned Here: T127734: Special characters in URL lead to redirect loop under IIS 7.5
T131414: Special characters in URL lead to redirect loop under Apache 2.2.22 on Debian 7
T112425: Pages with apostrophe character in their title aren't indexed by external search engines
T122786: Pages with apostrophe character in their title can't be translated by Google Translate
rMW155d555b83ec: MediaWiki.php: Redirect non-standard title urls to canonical
T105265: Redirecting ~ -> %7E causes a redirect loop in chrome

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I can view pages such as https://www.mediawiki.org/wiki/Manual:MediaWiki_Developer's_Guide or https://www.mediawiki.org/wiki/Manual:Chris_G's_botclasses, so I presume this fixed. @BBlack, should I file a separate bug about the Varnish stuff?

Ankry subscribed.Sep 10 2015, 1:42 PM

It seems it is not. In Firefox 33 and 36 the above pages still reports an incorrect redirect error.

I can't reproduce the error with Firefox 41 (current beta) on Windows 7, or any other modern browser, but I did reproduce it with Firefox 3.6, which still has a usage share on Wikimedia wikis comparable with Opera 12.

I see the loop in Firefox 39 too.

Nemo_bis renamed this task from Redirect loop in Opera 12 when the title contains a single quote to Redirect loop when the title contains a single quote (%27).Sep 10 2015, 3:11 PM

Nemo_bis set Security to None.

Nemo_bis added a parent task: T112069: Instability on fr.wikiversity project.

Nemo_bis raised the priority of this task from Lowest to Unbreak Now!.Sep 10 2015, 3:13 PM

Nemo_bis added a project: Browser-Support-Firefox.

Change 237401 had a related patch set uploaded (by Bartosz Dziewoński):
Revert "Do not encode "'" as ' (redirect loop in Opera 12)"

https://gerrit.wikimedia.org/r/237401

Change 237401 merged by jenkins-bot:
Revert "Do not encode "'" as ' (redirect loop in Opera 12)"

https://gerrit.wikimedia.org/r/237401

matmarex mentioned this in rMW8477b1b277c0: Revert "Do not encode "'" as %27 (redirect loop in Opera 12)".Sep 10 2015, 3:36 PM

Change 237405 had a related patch set uploaded (by Bartosz Dziewoński):
Revert "Do not encode "'" as ' (redirect loop in Opera 12)"

https://gerrit.wikimedia.org/r/237405

Just for the paper trail, new reports coming in: https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(technical)&oldid=680389124#Cookies_problem.3F

Change 237405 merged by jenkins-bot:
Revert "Do not encode "'" as ' (redirect loop in Opera 12)"

https://gerrit.wikimedia.org/r/237405

matmarex mentioned this in rMW7b3d7154fe4b: Revert "Do not encode "'" as %27 (redirect loop in Opera 12)".Sep 10 2015, 3:55 PM

Okay, sorry about this. So now we're back to the original state of Firefox working and Opera 12 not working. Restoring priority and stuff.

matmarex renamed this task from Redirect loop when the title contains a single quote (%27) to Redirect loop in Opera 12 when the title contains a single quote (%27).Sep 10 2015, 4:03 PM

matmarex lowered the priority of this task from Unbreak Now! to Lowest.

matmarex removed projects: Browser-Support-Firefox, MW-1.26-release, WMF-deploy-2015-09-08_(1.26wmf22), Patch-For-Review.

matmarex mentioned this in T112069: Instability on fr.wikiversity project.

matmarex removed a parent task: T112069: Instability on fr.wikiversity project.

In T106793#1572481, @BBlack wrote:

These will probably need matching changes in varnish here: https://github.com/wikimedia/operations-puppet/blob/production/modules/varnish/templates/vcl/wikimedia.vcl.erb#L262

In T106793#1626039, @matmarex wrote:

Okay, sorry about this. So now we're back to the original state of Firefox working and Opera 12 not working. Restoring priority and stuff.

If we can't come up with a single url that can be enforced, we should probably do what we originally did in Varnish and extend that config by merging these entries at the edge and thus support both variations. Since they appear to be mutually exclusive.

However that would be a Wikimedia-specific solution. It'd be nicer to build in tolerance at the MediaWiki level, so that third parties aren't stuck with inaccessible pages. Either as normal operation mode (e.g. allow both variations, and also purge both).

I'd prefer not to have this in Varnish as it keeps principle of authority on this matter within MediaWiki. And removing those from VCL would reduce complexity. However it does increase Varnish footprint slightly as it would no longer special-case these non-canonical copies.

Speaking of third parties, the other variations currently handled by Varnish presumably are also broken for some browsers viewing third-party wikis because we didn't use to enforce canonical urls by redirect. Third parties without a caching proxy had no issues. And those with were presumably serving stale cache potentially to some clients. But in 1.26 they'd be broken.

ReleaseTaggerBot added projects: MW-1.26-release, WMF-deploy-2015-09-15_(1.26wmf23), WMF-deploy-2015-09-08_(1.26wmf22).Sep 10 2015, 11:02 PM

Ciencia_Al_Poder mentioned this in T111933: Certain edit links on de.wp redirect to Main Page due to "NoScript".Sep 11 2015, 1:58 PM

matmarex removed projects: WMF-deploy-2015-09-08_(1.26wmf22), WMF-deploy-2015-09-15_(1.26wmf23), MW-1.26-release.Sep 12 2015, 5:12 PM

Ciencia_Al_Poder mentioned this in T112425: Pages with apostrophe character in their title aren't indexed by external search engines.Sep 13 2015, 9:55 AM

I'm not working on this currently. Unless more recent versions of Firefox can deal with unescaped apostrophes in URLs, it looks to me like the only solution would be to revert Krinkle's change rMW155d555b83eca6403e07d2094b074a8ed2f301ae and follow-ups, and I don't think we want to do that.

Krinkle mentioned this in T104755: Move URL-routing logic into MediaWiki.Oct 27 2015, 9:19 PM

Nemo_bis mentioned this in T116986: Set up UrlShortener dumps.Oct 30 2015, 3:28 PM

Schnark mentioned this in T122786: Pages with apostrophe character in their title can't be translated by Google Translate.Jan 4 2016, 9:23 AM

Krinkle renamed this task from Redirect loop in Opera 12 when the title contains a single quote (%27) to Pages with single quote in title are inaccessible by some clients (redirect loop).Jun 2 2016, 11:25 PM

Krinkle raised the priority of this task from Lowest to High.

Krinkle added a project: SEO.

Krinkle updated the task description. (Show Details)

Krinkle merged a task: T122786: Pages with apostrophe character in their title can't be translated by Google Translate.

Krinkle merged a task: T112425: Pages with apostrophe character in their title aren't indexed by external search engines.

Krinkle added subscribers: eranroz, Schnark.

Krinkle added a subscriber: StudiesWorld.

Krinkle added subscribers: Matanya, Tzafrir, TomerA and 4 others.

Krinkle added a subscriber: tstarling.

IKhitron added a subscriber: Yurik.Jun 3 2016, 12:05 AM

• MZMcBride subscribed.Jun 22 2016, 7:14 PM

There has been ongoing discussion about the indexing problem in https://productforums.google.com/d/msg/webmasters/wTPr_r_73sc/KyYsScvOnB4J

Yurik unsubscribed.Jun 26 2016, 10:22 AM

MW-1.27.

Error on browsers:
Vivaldi 1.2.490.43 "ERR_TOO_MANY_REDIRECTS"
Firefox 47.0
Firefox 47.0.1
Internet Explorer 11.0

I'm not sure which team in WMF is responsible for it, but I think Discovery. (If not please move to the correct team)

This is high important task resulting in search engines not indexing many new pages on Wikipedia.

Hi @eranroz and @RogueFiber - please post the URL that you're using to test with.

I did a quick test on Chrome, Opera and Firefox and I'm not seeing any errors or weirdness using the O'Hare URLs listed above on en.wiki and fr.wiki.

For example:
https://he.wikipedia.org/wiki/%D7%A1%D7%95%D7%A4%D7%99_%D7%A6%27%D7%A8%D7%A0%D7%99%D7%90%D7%A7

This page isn't indexed by google though newer pages are already indexed.

Ah, I see now - Google does this (did you mean) but that is less than optimal.

google-indexing_issue--סופי_צ%27רניאק.png (363×827 px, 59 KB)

We don't have a direct relationship with Google to help them index (or do anything else) with Wiki pages, that I know of. I'll send this to our Discovery mailing list and see if anyone can help.

So...interestingly enough. As I was writing up an email about this, I checked again on Google and now - it's working! Maybe we just needed to wait a bit longer as their spiders gather all the new pages?

google-indexing_issue--resolved.png (722×790 px, 227 KB)

RogueFiber added a comment.Aug 17 2016, 5:59 PM

This comment was removed by RogueFiber.

RogueFiber added a comment.Aug 17 2016, 6:04 PM

This comment was removed by RogueFiber.

Krinkle updated the task description. (Show Details)Sep 9 2016, 9:45 PM

Change 309575 had a related patch set uploaded (by Paladox):
Revert "MediaWiki.php: Redirect non-standard title urls to canonical"

https://gerrit.wikimedia.org/r/309575

gerritbot added a project: Patch-For-Review.Sep 9 2016, 9:46 PM

Krinkle removed a project: Patch-For-Review.Sep 10 2016, 12:47 AM

In T106793#1759378, @matmarex wrote:

I'm not working on this currently. Unless more recent versions of Firefox can deal with unescaped apostrophes in URLs, it looks to me like the only solution would be to revert Krinkle's change rMW155d555b83eca6403e07d2094b074a8ed2f301ae and follow-ups, and I don't think we want to do that.

It appears to me like this is still the only option since it looks to me that nobody involved here has a clue how to fix it. I am stuck at MW 1.25 with not upgrade path due to the issues this change causes.

I would like to update from unsupported MW 1.25. Any news since I still have no upgrade path.

In T106793#2654001, @Kghbln wrote:

I am stuck at MW 1.25 with not upgrade path due to the issues this change causes.

@Kghbln: Which browsers and browser versions are affected in your case?

RogueFiber unsubscribed.Oct 17 2016, 2:39 PM

Currently it is Firfox 49, Chrome 53, Opera 12.26 (on Linux Mint 17.3) and Edge 25 and Internet Explorer 11 (on Win 10), so basically everywhere.

Okay, per earlier comments, I am merging the revert (and backporting to MW 1.28). I think the detailed task description here is sufficient justification, and proponents of the original change seem unable or unwilling to fix the problems.

If the redirecting functionality is to be reimplemented, it should probably be researched more carefully… (Or, if we at Wikimedia decide that the pain suffered by some users is worth the gain in "cleanliness", it should be gated behind a configuration option.)

matmarex merged tasks: T127734: Special characters in URL lead to redirect loop under IIS 7.5, T131414: Special characters in URL lead to redirect loop under Apache 2.2.22 on Debian 7.Nov 9 2016, 1:14 AM

matmarex added subscribers: PeterJCLaw, TerraCodes, Luke081515 and 6 others.

Change 309575 merged by jenkins-bot:
Revert "MediaWiki.php: Redirect non-standard title urls to canonical"

https://gerrit.wikimedia.org/r/309575

Change 320559 had a related patch set uploaded (by Bartosz Dziewoński):
Revert "MediaWiki.php: Redirect non-standard title urls to canonical"

https://gerrit.wikimedia.org/r/320559

Change 320559 merged by Bartosz Dziewoński:
Revert "MediaWiki.php: Redirect non-standard title urls to canonical"

https://gerrit.wikimedia.org/r/320559

matmarex closed this task as Resolved.Nov 9 2016, 1:28 AM

matmarex claimed this task.

matmarex mentioned this in T67402: URLs for the same title without extra query parameters should have the same canonical link.

matmarex removed a project: Patch-For-Review.

ReleaseTaggerBot added projects: MW-1.28-release-notes, MW-1.29-release-notes, MW-1.29-release (WMF-deploy-2016-11-15_(1.29.0-wmf.3)).Nov 9 2016, 2:00 AM

@matmarex Thanks for dealing with this issue. I think it would also make sense to backport to REL1_27

Change 320795 had a related patch set uploaded (by Bartosz Dziewoński):
Revert "MediaWiki.php: Redirect non-standard title urls to canonical"

https://gerrit.wikimedia.org/r/320795

gerritbot added a project: Patch-For-Review.Nov 10 2016, 5:11 PM

Change 320795 merged by jenkins-bot:
Revert "MediaWiki.php: Redirect non-standard title urls to canonical"

https://gerrit.wikimedia.org/r/320795

Also backported to REL1_27. The change should be included in the upcoming MediaWiki 1.27.2 release, unfortunately I don't know when it's going to be (I don't think it's planned yet).

matmarex edited projects, added MW-1.27-release-notes; removed Patch-For-Review.Nov 10 2016, 5:21 PM

@matmarex Thanks a lot. Very much appreciated! Time to move on from 1.25 to 1.27 as soon as 1.27.2 is out. :)

Nemo_bis mentioned this in T144100: Pageview dumps incorrectly formatted, need to escape special characters.Dec 8 2016, 8:16 AM

Nemo_bis mentioned this in T120085: RFC: Serve Main Page of Wikimedia wikis from a consistent URL.Jan 22 2017, 8:08 AM

• Deskana moved this task from Tag to Done on the SEO board.Jul 6 2018, 10:28 AM

matmarex mentioned this in T257966: URL shall be terminated by %2E or _ if page name ends with dot.Jul 15 2020, 1:43 PM

Krinkle mentioned this in T276173: Unable to share link to article via Apple Message if title ends with exclamation mark.Mar 1 2021, 9:06 PM

Krinkle mentioned this in T138093: Investigate query parameter normalization for MW/services.Aug 3 2022, 11:48 PM

	F4365770: google-indexing_issue--resolved.png
	Aug 15 2016, 11:26 PM

	F4365709: google-indexing_issue--סופי_צ%27רניאק.png
	Aug 15 2016, 10:51 PM

	F2565355: pasted_file
	Sep 10 2015, 3:08 PM

Pages with single quote in title are inaccessible by some clients (redirect loop)Closed, ResolvedPublicActions