Page MenuHomePhabricator
Create Task
Maniphest T17601

robots.txt should be split to common and localisable parts
Closed, ResolvedPublic
Actions

Description

Currently, if [[MediaWiki:Robots.txt]] exists, it is used as this wiki's robots.txt. However, this method blocks the way for developers to introduce centralised changes to robots.txt, such as blocking of new spiders. Therefore, I propose to build it from two parts: centralised one, containing user agent rules and prohibition from indexing /w/, and content of MediaWiki:Robots.txt, if present. Centralised part could be accessible to shell users, or be mirrored from Meta, like www portals currently do.

Version: unspecified
Severity: enhancement
URL: http://en.wikipedia.org/robots.txt

Details

Reference
bz15601

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 21 2014, 10:21 PM
• bzimport added a project: WMF-General-or-Unknown.
• bzimport set Reference to bz15601.
MaxSem created this task.Sep 14 2008, 9:25 AM
• bzimport added a comment.Sep 15 2008, 9:23 PM

lilewyn wrote:

http://meta.wikimedia.org/wiki/Template:Robots.txt would (imo) be a logical location.

• bzimport added a comment.Sep 15 2008, 9:25 PM

mike.lifeguard+bugs wrote:

No, it would likely be located at the default location, not on-wiki. The whole point is that the common part will be set by sysadmins, not end-users. They can edit the per-wiki part at MediaWiki:Robots.txt already.

• bzimport added a comment.Sep 17 2008, 7:26 PM

jeluf wrote:

Done.

• MZMcBride added a comment.Feb 25 2013, 9:46 AM

For reference, this is documented at [[wikitech:robots.txt]].

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits