Page MenuHomePhabricator

prevent search engines from indexing userspace on enwiki
Closed, ResolvedPublic

Description

There is a consensus at the Village Pump that all pages in userspace should opt out of being indexed by search engines.

Event Timeline

Mdann52 raised the priority of this task from to Needs Triage.
Mdann52 updated the task description. (Show Details)
Mdann52 subscribed.
Mdann52 claimed this task.

Looks like it can be handled locally, closing.

Actually, while robots.txt can do this, having it native in the MW config is probably a better solution

Generally this sounds fine to me, but please hold off on merging any changes that implement this for now, as I'm performing a quick investigation to see if there's some other cause for this.

See here: https://lists.wikimedia.org/pipermail/wikitech-l/2015-July/082314.html

Dereckson subscribed.

Assigned to @Deskana pending investigation.

I'm a little skeptical that a 2-week long discussion about NOINDEXing an entire namespace can be considered consensus, especially as this seems to be a perennial topic.

On the point of investigations, user talk space has been NOINDEXed since 2008; see also T15890.

Deskana set Security to None.

Unassigning myself, as I never did find a root cause for this.

Change 237330 had a related patch set uploaded (by Mdann52):
noindex userspace, per T104797

https://gerrit.wikimedia.org/r/237330

Change 237330 had a related patch set uploaded (by Alex Monk):
noindex user namespace on en.wikipedia.org

https://gerrit.wikimedia.org/r/237330

@Deskana: Is this good to go now?

I suppose so; I have no particularly strong feelings about this task myself, but it seems logical enough.

I figure we can try it, and revert if there is some issue caused by it. Let's add the User-notice tag and ping @Johan so that he can put it into Tech News; that way, people know to come here and ask for a revert if there's an issue.

This only concerns English Wikipedia, right? We try to avoid putting things that are only relevant to one wiki into Tech News as it's translated into 13–15 languages and goes out to a fair number of wikis, but I could manually add an update about this to all English Wikipedia community pages reached by Tech News when I send it out on Monday. Makes sense?

Sounds fine to me. What pages does that include? Is there a distribution list somewhere?

On English Wikipedia, that would be the technical village pump and Wikipedia:Tech news. The distribution list can be found here:
https://meta.wikimedia.org/wiki/Global_message_delivery/Targets/Tech_ambassadors#Community_pages

On English Wikipedia, that would be the technical village pump and Wikipedia:Tech news. The distribution list can be found here:
https://meta.wikimedia.org/wiki/Global_message_delivery/Targets/Tech_ambassadors#Community_pages

Thanks for announcing this today, Johan.

I think we're good to merge this config change now.

Nobody merged the patch, even though it was announced...

Nobody merged the patch, even though it was announced...

I could try to get it done on the next SWAT window but I am not sure whether it would be accepted on this week or the next. Since it's a simple config change, it might be allowed though.

Change 237330 merged by jenkins-bot:
noindex user namespace on en.wikipedia.org

https://gerrit.wikimedia.org/r/237330

Change has been deployed and the meta robots tag is now present on user ns pages. It could take anything from a few minutes to weeks for the changes to take effect on the search results though.