Page MenuHomePhabricator

Add charset to robots.txt
Closed, ResolvedPublic

Description

From mediawiki-l discussion about not viewing robots utf characters http://thread.gmane.org/gmane.org.wikimedia.mediawiki/33601

robots.txt served from wikimedia wikis should have content-type text/plain;charset=utf-8 not just text/plain

robots.txt spec doesn't take content-types into account, no bot should be relying on a specific content-type.
http://www.robotstxt.org/orig.html


Version: unspecified
Severity: minor
URL: http://thread.gmane.org/gmane.org.wikimedia.mediawiki/33601

Details

Reference
bz22932

Related Objects

StatusSubtypeAssignedTask
OpenNone
ResolvedNone

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 11:00 PM
bzimport set Reference to bz22932.
bzimport added a subscriber: Unknown Object (MLST).

Now it is Content-Type: text/html; charset=utf-8
The mime should have remained text/plain, not changed to text/html.

The requested Content-Type is: text/plain; charset=utf-8

jeluf wrote:

Grmpf. Cut&Paste-Error, please excuse.

Should be fixed now.