Page MenuHomePhabricator
Create Task
Maniphest T13508

Correct syntax in robots.txt
Closed, ResolvedPublic
Actions

Description

Author: ais523

Description:
corrected version of robots.txt (unfortunately with a BOM at the start I couldn't get rid of)

It turns out that there are some syntax errors in Wikimedia's robots.txt file; there are some leading slashes missing, and some misplaced blank lines. I've come up with a corrected version, with the formatting made more consistent, the leading slashes added, and the blank lines commented out.

Unfortunately, I'm on a public computer at the moment on which I can't install software, and that's running Windows. So the diff I created is in the wrong format and shows no context, and both the diff and the new file have a Byte Order Mark. (The diff even shows the addition of the BOM; this needs to be removed before the patch is applied!) Anyone could submit a corrected version of the patch, which would make it easier for shell developers to apply.

Version: unspecified
Severity: enhancement
URL: http://en.wikipedia.org/robots.txt

Attached:

robots2.txt13 KBDownload

Details

Reference
bz11508

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 21 2014, 9:56 PM
• bzimport added projects: WMF-General-or-Unknown, Shell.
• bzimport set Reference to bz11508.
• bzimport added a subscriber: Unknown Object (MLST).
• bzimport created this task.Sep 30 2007, 10:14 AM
• bzimport added a comment.Sep 30 2007, 10:15 AM

ais523 wrote:

patch in the wrong format that suggests adding a BOM at the start

attachment robots.diff ignored as obsolete

• bzimport added a comment.Sep 30 2007, 10:16 AM

ais523 wrote:

patch in the wrong format that suggests adding a BOM at the start

(attachment 4181 didn't upload properly, maybe because it was in the wrong format, trying again with a different content type)

attachment robots.diff ignored as obsolete

• bzimport added a comment.Sep 30 2007, 10:18 AM

ais523 wrote:

Comment on attachment 4182
patch in the wrong format that suggests adding a BOM at the start

Sorry, the diffing seemed to go wrong (attachment 4182 is also incorrect, and it seems the same thing happened to my source file); could someone else diff the current robots.txt with my version to produce the patch?

• bzimport added a comment.Sep 30 2007, 11:51 AM

webboy wrote:

Patch for robots.txt

Diff between http://en.wikipedia.org/robots.txt and attachment 4180

Attached:

robotstxt.patch7 KBDownload

• brooke added a comment.Oct 1 2007, 8:06 PM

Applied, thanks!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits