What about URLs like title=Special:GoToLinkedPage&site=dewiki&itemid=Q123456?

You'll probably want to consider adapting the extension to make to enforce this in a better way for all users..

Perhaps the X-Robots-Tag http header can be used to remove indexing of the redirect... Not sure, redirects can be a bit problematic in that way.

Right, this will still be a problem on Wikibase installations other than wikidata.org. How does MediaWiki core do this?

Google still index these pages: https://www.google.com/search?q=African+wild+dog+site:wikidata.org (notice that the first result has a cached version of yesterday)

In T144308#2662636, @Sjoerddebruin wrote:

Google still index these pages: https://www.google.nl/search?client=safari&rls=en&q=African+wild+dog+site:wikidata.org&ie=UTF-8&oe=UTF-8&gfe_rd=cr&ei=G1HlV9TLHemRwAKeurXICg (notice that the first result has a cached version of yesterday)

Probably due to the trailing slashes in the request posted in T144308#2597079. The URL for the cached entry is http://www.wikidata.org/wiki/Special:GoToLinkedPage?site=enwiki&itemid=Q173651. (Which has no slash after GoToLinkedPage)

@thiemowmde: Maybe we should remove the trailing slashes in the robots.txt?

FYI: you disallowed crawling, that doesn't mean you disallowed indexing for modern search engines. If another indexed page links to the url, that google will still index it.

To quote

When you block URLs from being indexed in Google via robots.txt, they may still show those pages as URL only listings in their search results. A better solution for completely blocking the index of a particular page is to use a robots noindex meta tag on a per page bases. You can tell them to not index a page, or to not index a page and to not follow outbound links by inserting either of the following code bits in the HTML head of your document that you do not want indexed.

This why we have NOINDEX and setRobotPolicy on OutputPage etc.. it's just that these are redirects and/or not necessarily HTML. That's why i pointed at X-Robots-Tag.
Another thing to pay attention to, is indicating canonical urls whenever possible

As you said, Special:GoToLinkedPage redirects and does not output HTML (except for the form, which is a single page, and the reason why Disallow: /wiki/Special:GoToLinkedPage should not be used). The target pages of these redirects are Wikipedia articles. They should be indexed, and they already have canonical tags.

I'm not sure what an X-Robots-Tag will do when used with a redirect.

• I believe there is no point in crawling Special:GoToLinkedPage URLs, because they are guaranteed to do nothing but redirect to Wikipedia articles. We know each redirect represents a sitelink, and sitelinks are already accessible on the ordinary item page. We know all this. Google does not.
• Similar for Special:ItemByTitle, which redirects to a Wikidata item. The exact same links already exist in the sidebars of the connected Wikipedia articles.

@Mbch331, please add the following lines to https://www.wikidata.org/wiki/MediaWiki:Robots.txt:

Disallow: /wiki/Special:GoToLinkedPage?
Disallow: /wiki/Special:ItemByTitle?
Disallow: /wiki/Special:SetSiteLink?

Do not remove the slashes, because this would exclude the special page forms itself. We want these to appear in a Google search.

Request on https://www.wikidata.org/wiki/MediaWiki_talk:Robots.txt#Also_exclude_URLs_with_question_marks is done.

https://www.google.com/search?q=Ethiopian+wolf+site%3Awikidata.org was visited yesterday and is still indexed by Google.

We changed our robots.txt two and a half weeks ago. Re-visiting possibly millions of URLs in such a short time is something neither we nor Google want. At the moment there are 28,000 left, it seems.

The links we want to exclude are tools and never meant to be indexed. On the one hand, Google can't know this. On the other hand, I wonder why an existing canonical tag is basically ignored and Google acts like it found Wikipedia articles on Wikidata.

Let's check again in another two weeks.