Page MenuHomePhabricator
Create Task
Maniphest T148994

Links with `?oldformat=true` or escaped slashes bypass NOINDEX
Closed, ResolvedPublic
Actions

Description

Articles for Deletion discussions on the English Wikipedia, in theory, are NOINDEX-tagged so they don't show up in search engines. However, the same link, encoded with ?oldformat=true at the end, still does. This negates the NOINDEX tag somewhat.

For example, a search for foo will _not_ come up with https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Foo but will for https://en.wikipedia.org/wiki/Wikipedia%3AArticles_for_deletion%2FFoo?oldformat=true.

See, for example, the second page of Google search results for "Todd Claydon".

This may well be something unfixable (it appears that Google is picking up on the URL itself).

Related Objects
Search...

Event Timeline

jrbs created this task.Oct 24 2016, 5:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 24 2016, 5:52 PM
Nick subscribed.Oct 24 2016, 5:57 PM
jrbs added a parent task: T6776: Add entries to robots.txt to block AFD/VFD/MFD.Oct 24 2016, 6:05 PM
MarkTraceur subscribed.Dec 8 2016, 4:31 PM

Just asked for a new disallow rule at https://en.wikipedia.org/wiki/MediaWiki_talk:Robots.txt#Google_thinks_it.27s_cute.2C_we_need_to_blacklist_Wikipedia.253AArticles_for_deletion.252F - seemed relevant to this discussion.

Should we perhaps consider not parsing article paths as escaped query string parameters, if the Location header has no query string?

jrbs renamed this task from Links with `?oldformat=true` bypass NOINDEX somehow to Links with `?oldformat=true` or escaped slashes bypass NOINDEX.Dec 8 2016, 6:31 PM
MarkTraceur unsubscribed.Jan 27 2017, 10:36 PM
Liuxinyu970226 subscribed.Sep 2 2018, 9:31 AM
Liuxinyu970226 edited projects, added: MediaWiki-Parser; removed: WMF-General-or-Unknown.Dec 23 2018, 11:18 AM
Krinkle closed this task as Resolved.Nov 11 2025, 3:22 AM
Krinkle claimed this task.
Krinkle added projects: SEO, MediaWiki-Core-Platform-Team (Radar).
Krinkle added subscribers: MarkTraceur, Krinkle.
In T148994#2857266, @MarkTraceur wrote:

Just asked for a new disallow rule at https://en.wikipedia.org/wiki/MediaWiki_talk:Robots.txt#Google_thinks_it.27s_cute.2C_we_need_to_blacklist_Wikipedia.253AArticles_for_deletion.252F - seemed relevant to this discussion.

Shortly after the above talk page thread, in Jan 2017, Legokgtm changed/added (diff) the robots.txt entry to omit the trailing slash.

- Disallow: /wiki/Wikipedia:Articles_for_deletion/
+ Disallow: /wiki/Wikipedia:Articles_for_deletion
- Disallow: /wiki/Wikipedia%3AArticles_for_deletion/
+ Disallow: /wiki/Wikipedia%3AArticles_for_deletion

When I inspect https://en.wikipedia.org/wiki/Wikipedia%3AArticles_for_deletion%2FFoo?oldformat=true in the Google Seach Console, it now says it is blocked by robots.txt.

Searching "Todd Claydon" site:wikipedia.org on google.com today yields no results.

See also T12288: Add talk pages of deletion discussions to robots.txt

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits