Google (and others) is indexing data dumps
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	555
	Oct 20 2007, 11:41 PM

Description

Probably the absense of a robots.txt file at download.wikimedia.org is a expected behaviour, but it is resulting in bandwidth waste: Google is indexing the .gz files, Yahoo the .xml ones

http://www.google.com/search?q=some+site%3Adownload.wikimedia.org
http://search.yahoo.com/search?p=some+site%3Adownload.wikimedia.org

Version: unspecified
Severity: enhancement
URL: http://download.wikimedia.org/robots.txt

Details

Reference: bz11720

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Invalid		None	T19004 [DO NOT USE] Search engine indexing (tracking)
		Declined		• Tfinc	T13720 Google (and others) is indexing data dumps

Event Timeline

• bzimport raised the priority of this task from to Low.Nov 21 2014, 9:57 PM

• bzimport added projects: Datasets-General-or-Unknown, Shell.

• bzimport set Reference to bz11720.

555 created this task.Oct 20 2007, 11:41 PM

Tomasz, this may or may not be something we want to adjust. :)

ArielGlenn moved this task from Backlog to Done on the Datasets-General-or-Unknown board.Jul 14 2016, 10:17 PM

Google (and others) is indexing data dumpsClosed, DeclinedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Google (and others) is indexing data dumps
Closed, DeclinedPublic
Actions

Related Objects
Search...