Page MenuHomePhabricator

disallowing robots on talk pages and deletion and "meta" talk pages pages in de.wikipedia
Closed, ResolvedPublic


Author: arnomane

Please add the content of the following attached file for de.wikipedia robots.txt under "User-agent: *"
The disallow for talk pages was voted in a large vote (seeür_Nichtindizierung_von_Diskussionsseiten_durch_Suchmaschinen )
and would help very much with ongoing highly problematic discussions about topics that are currently in
the media and in court. I don't know if the "talk"-equivalent is necessary. I added it as you can reach
inside every local wiki the namespaces via their english name and you can ignore it if it is not needed
in order to make the "noindex" perfect.

Version: unspecified
Severity: normal
Platform: Other



Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 9:04 PM
bzimport set Reference to bz4937.
bzimport added a subscriber: Unknown Object (MLST).

arnomane wrote:

additional lines the de.wikipedia robots.txt


The format is wrong in the attachment, but that's a detail.

I'll set up the vfd bits in robots.txt, and see about the talk pages separately.

arnomane wrote:

I have read the robots.txt page on meta so I guess the * are not necessary, right? Concerning the talk pages there was at first another
idea (also documented at the web link above) with adding <meta name="robots" content="noindex,follow" /> in every weg page via a change in
the MediaWiki software as it is with the "nofollow" for weblinks.

But a solution with robots.txt would avoid that this page gets even loaded so it would reduce the requests by web spiders.

wikipedia wrote:

Please add
"Disallow: /wiki/Wikipedia:Vandalismusmeldung/"
below the line
"Disallow: /wiki/Wikipedia:Vandalensperrung/"
in robots.txt

Please also add the line
"Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/"
in the same paragraph.

shagge wrote:

As Tims request was not yet fullfilled and we had another discussion about this topic on OTRS-de-l we would kindly like to ask you to please at least add

Disallow: /wiki/Wikipedia:Vandalismusmeldung/

Besides that the following paths should be added:

Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/
Disallow: /wiki/Wikipedia:Löschprüfung/
Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/
Disallow: /wiki/Wikipedia:Administratoren/Notizen/
Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/

(In reply to comment #6)

Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/
Disallow: /wiki/Wikipedia:Löschprüfung/
Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/
Disallow: /wiki/Wikipedia:Administratoren/Notizen/
Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/

Correction of the the 3rd line (Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/)

Disallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung

jeluf wrote:

Please don't reopen bugs to request new things. Those bugs get lost within all these DPL requests.