Page MenuHomePhabricator

MediaWiki:Robots.txt has no effect on Commons
Closed, ResolvedPublic

Description

The content of [[commons:MediaWiki:Robots.txt]] does not seem to show up at http://commons.wikimedia.org/robots.txt . It should, shouldn't it? Is something wrong with the configuration?


Version: unspecified
Severity: normal
URL: http://commons.wikimedia.org/robots.txt

Details

Reference
bz22040

Related Objects

StatusSubtypeAssignedTask
OpenNone
ResolvedNone

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 10:53 PM
bzimport set Reference to bz22040.
bzimport added a subscriber: Unknown Object (MLST).

It seems [[meta:MediaWiki:Robots.txt]] isn't either. Perhaps because they have a pedia suffix?

Yeah, looks like the relevant RewriteRules are only present in http://noc.wikimedia.org/conf/main.conf but not in http://noc.wikimedia.org/conf/remnant.conf where Commons and Meta are. Sprinkling a few suitably placed copies of

  1. Make robots.txt editable via Mediawiki:robots.txt RewriteRule ^/robots.txt$ /w/robots.php [L]

into that file would seem likely to fix it. (Also should probably do the same for the other relevant *.conf files.)

CCing JeLuF on this -- he fixed bug 15663, so I assume he knows something about this robots.txt setup. ;)

Those files have interesting data, like the non-working wikipedia.geo.blitzed.org ServerAlias, or the robots file name. Why isn't robots.php in svn?

jeluf wrote:

Fixed config for meta and commons.
Added robots.php to SVN.