Page MenuHomePhabricator

Robots.txt incomplete
Closed, ResolvedPublic

Description

The file robots.txt to http://static.wikipedia.org/ is incomplete...


Version: unspecified
Severity: normal
URL: http://static.wikimedia.org/robots.txt

Details

Reference
bz4275

Related Objects

StatusSubtypeAssignedTask
OpenNone
ResolvedNone

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 9:00 PM
bzimport set Reference to bz4275.
bzimport added a subscriber: Unknown Object (MLST).

robchur wrote:

...in what way? And what issues does this cause?

The domain http://static.wikipedia.org/ have lots of directories listeds in your
main page, but the robots.txt disallow bots to download data only to 9 of them.

Fixed it thanks for the report.