Flow does not follow $wgNamespaceRobotPolicies
Closed, ResolvedPublic

Description

French Wikipedia has a decided in 2010 to not index users pages and user talk pages. This has been setup on the global preferences (iirc). But user talk pages using Flow boards are indexed.

Examples of Google searches:

Trizek-WMF updated the task description. (Show Details)
Trizek-WMF raised the priority of this task from to High.
Trizek-WMF added a subscriber: Trizek-WMF.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Mattflaschen-WMF renamed this task from Global preferences concerning talk pages indexation are not respected by Flow to Flow does not follow $wgNamespaceRobotPolicies.Dec 22 2015, 10:50 PM
Mattflaschen-WMF set Security to None.
Framawiki added a subscriber: Framawiki.

Change 403416 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/Flow@master] Implement robot-policy

https://gerrit.wikimedia.org/r/403416

Change 403416 merged by jenkins-bot:
[mediawiki/extensions/Flow@master] Implement robot-policy

https://gerrit.wikimedia.org/r/403416

Hey, we're moving forward, thanks @SBisson !

Checked in betalabs - User and User talk (SD and non-SD pages) are not found by google searches.

@SBisson two questions:
(1)

French Wikipedia has a decided in 2010 to not index users pages and user talk pages.

Does it mean that only French wikipedia has such a setup? And now enwiki betalabs is following that setup also?

(2) Google search cannot find any SD topic (hidden or not-hidden) in enwiki betalabs. Is it a special setup for betalabs? Nothing in betalabs gets indexed?

Checked in betalabs - User and User talk (SD and non-SD pages) are not found by google searches.

@SBisson two questions:
(1)

French Wikipedia has a decided in 2010 to not index users pages and user talk pages.

Does it mean that only French wikipedia has such a setup?

I count 18 wikis (including frwiki, enwiki, dewiki) that have decided to disable indexing on NS_USER and/or NS_USER_TALK.

And now enwiki betalabs is following that setup also?
(2) Google search cannot find any SD topic (hidden or not-hidden) in enwiki betalabs. Is it a special setup for betalabs? Nothing in betalabs gets indexed?

It makes sense that we don't want people searching google to land on betalabs but I don't know how it's configured. I don't see it in the config repo.

thx, @SBisson - I will check it after deployment.

Checked in frwiki (wmf.17)

SD User talk pages and non-SD have now "noindex,follow". However, it may take some time for de-indexing the pages.

Etonkovidova closed this task as Resolved.Jan 18 2018, 11:36 PM