rest.wikimedia.org is crawled by search engines
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Nemo_bis
	Nov 28 2015, 9:42 AM

Project Tags

Referenced Files

None

Subscribers

Description

I find pages such as https://rest.wikimedia.org/fr.wikipedia.org/v1/page/html/WebEx in Google search results. (Yes, only if I specify site:wikimedia.org.)

There are at least two severe bugs:

https://rest.wikimedia.org/robots.txt should exclude crawling by search engines;
the canonical URL of each page should be set.

		Status	Subtype	Assigned	Task
		Invalid	Feature	None	T16720 robots.txt (tracking)
		Resolved		• GWicke	T119786 rest.wikimedia.org is crawled by search engines

Nemo_bis raised the priority of this task from to High.

Nemo_bis updated the task description. (Show Details)

Nemo_bis added projects: SEO, RESTBase.

Nemo_bis subscribed.

@Nemo_bis, thanks for the report!

I still don't see canonical URLs though, hence these pages fragment our PageRank and contribute to the decline of Wikimedia projects.

@Nemo_bis: The current robots.txt basically hides API content from search engines completely. There is no effect of the API on PageRank at all.