Page MenuHomePhabricator
Create Task
Maniphest T6275

Robots.txt incomplete
Closed, ResolvedPublic
Actions

Description

The file robots.txt to http://static.wikipedia.org/ is incomplete...

Version: unspecified
Severity: normal
URL: http://static.wikimedia.org/robots.txt

Details

Reference
bz4275

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenFeatureNoneT16720 robots.txt (tracking)
 ResolvedNoneT6275 Robots.txt incomplete

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 21 2014, 9:00 PM
• bzimport added a project: Datasets-General-or-Unknown.
• bzimport set Reference to bz4275.
• bzimport added a subscriber: Unknown Object (MLST).
555 created this task.Dec 14 2005, 3:37 PM
• bzimport added a comment.Dec 14 2005, 3:48 PM

robchur wrote:

...in what way? And what issues does this cause?

555 added a comment.Dec 14 2005, 3:56 PM

The domain http://static.wikipedia.org/ have lots of directories listeds in your
main page, but the robots.txt disallow bots to download data only to 9 of them.

hashar added a comment.Dec 14 2005, 8:32 PM

Fixed it thanks for the report.

ArielGlenn moved this task from Backlog to Done on the Datasets-General-or-Unknown board.Jul 12 2016, 6:43 AM
Liuxinyu970226 added a parent task: T16720: robots.txt (tracking).Mar 1 2019, 1:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits