Page MenuHomePhabricator

prevent search engines from indexing userspace on enwiki
Closed, ResolvedPublic

Description

There is a consensus at the Village Pump that all pages in userspace should opt out of being indexed by search engines.

Event Timeline

Mdann52 created this task.Jul 5 2015, 10:33 AM
Mdann52 raised the priority of this task from to Needs Triage.
Mdann52 updated the task description. (Show Details)
Mdann52 added a subscriber: Mdann52.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptJul 5 2015, 10:33 AM
Mdann52 closed this task as Resolved.Jul 5 2015, 10:37 AM
Mdann52 claimed this task.

Looks like it can be handled locally, closing.

Mdann52 reopened this task as Open.Jul 5 2015, 10:43 AM

Actually, while robots.txt can do this, having it native in the MW config is probably a better solution

Mdann52 removed Mdann52 as the assignee of this task.Jul 5 2015, 10:43 AM
Deskana added a subscriber: Deskana.Jul 6 2015, 5:32 PM

Generally this sounds fine to me, but please hold off on merging any changes that implement this for now, as I'm performing a quick investigation to see if there's some other cause for this.

See here: https://lists.wikimedia.org/pipermail/wikitech-l/2015-July/082314.html

Dereckson added a subscriber: Dereckson.

Assigned to @Deskana pending investigation.

Izno added a subscriber: Izno.Jul 17 2015, 8:01 PM

I'm a little skeptical that a 2-week long discussion about NOINDEXing an entire namespace can be considered consensus, especially as this seems to be a perennial topic.

On the point of investigations, user talk space has been NOINDEXed since 2008; see also T15890.

Deskana removed Deskana as the assignee of this task.Aug 15 2015, 2:08 AM
Deskana set Security to None.

Unassigning myself, as I never did find a root cause for this.

Change 237330 had a related patch set uploaded (by Mdann52):
noindex userspace, per T104797

https://gerrit.wikimedia.org/r/237330

Mdann52 updated the task description. (Show Details)Sep 10 2015, 5:51 AM
Mdann52 claimed this task.Sep 10 2015, 9:12 AM

Working on this

Change 237330 had a related patch set uploaded (by Alex Monk):
noindex user namespace on en.wikipedia.org

https://gerrit.wikimedia.org/r/237330

NeilN added a subscriber: NeilN.Sep 14 2015, 5:19 PM
Krenair added a subscriber: Krenair.

@Deskana: Is this good to go now?

Deskana added a subscriber: Johan.Sep 18 2015, 4:07 AM

@Deskana: Is this good to go now?

I suppose so; I have no particularly strong feelings about this task myself, but it seems logical enough.

I figure we can try it, and revert if there is some issue caused by it. Let's add the User-notice tag and ping @Johan so that he can put it into Tech News; that way, people know to come here and ask for a revert if there's an issue.

Johan added a comment.Sep 18 2015, 3:58 PM

This only concerns English Wikipedia, right? We try to avoid putting things that are only relevant to one wiki into Tech News as it's translated into 13–15 languages and goes out to a fair number of wikis, but I could manually add an update about this to all English Wikipedia community pages reached by Tech News when I send it out on Monday. Makes sense?

Sounds fine to me. What pages does that include? Is there a distribution list somewhere?

Johan added a comment.Sep 18 2015, 5:43 PM

On English Wikipedia, that would be the technical village pump and Wikipedia:Tech news. The distribution list can be found here:
https://meta.wikimedia.org/wiki/Global_message_delivery/Targets/Tech_ambassadors#Community_pages

On English Wikipedia, that would be the technical village pump and Wikipedia:Tech news. The distribution list can be found here:
https://meta.wikimedia.org/wiki/Global_message_delivery/Targets/Tech_ambassadors#Community_pages

Thanks for announcing this today, Johan.

I think we're good to merge this config change now.

Deskana moved this task from Needs triage to Tracking on the Discovery board.Sep 22 2015, 5:27 PM
TheDJ added a subscriber: TheDJ.Nov 11 2015, 8:37 PM

Nobody merged the patch, even though it was announced...

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 11 2015, 8:37 PM
TheDJ awarded a token.Nov 23 2015, 7:39 AM

Nobody merged the patch, even though it was announced...

I could try to get it done on the next SWAT window but I am not sure whether it would be accepted on this week or the next. Since it's a simple config change, it might be allowed though.

Change 237330 merged by jenkins-bot:
noindex user namespace on en.wikipedia.org

https://gerrit.wikimedia.org/r/237330

Change has been deployed and the meta robots tag is now present on user ns pages. It could take anything from a few minutes to weeks for the changes to take effect on the search results though.

Glaisher closed this task as Resolved.Nov 23 2015, 4:50 PM
Restricted Application added a subscriber: JEumerus. · View Herald TranscriptFeb 10 2016, 6:52 AM