Add charset to robots.txt
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	Platonides
	Mar 23 2010, 5:03 PM

Description

From mediawiki-l discussion about not viewing robots utf characters http://thread.gmane.org/gmane.org.wikimedia.mediawiki/33601

robots.txt served from wikimedia wikis should have content-type text/plain;charset=utf-8 not just text/plain

robots.txt spec doesn't take content-types into account, no bot should be relying on a specific content-type.
http://www.robotstxt.org/orig.html

Version: unspecified
Severity: minor
URL: http://thread.gmane.org/gmane.org.wikimedia.mediawiki/33601

		Status	Subtype	Assigned	Task
		Open	Feature	None	T16720 robots.txt (tracking)
		Resolved		None	T24932 Add charset to robots.txt

• bzimport raised the priority of this task from to Medium.Nov 21 2014, 11:00 PM

• bzimport set Reference to bz22932.

• bzimport added a subscriber: Unknown Object (MLST).

jeluf wrote:

Done.

Now it is Content-Type: text/html; charset=utf-8
The mime should have remained text/plain, not changed to text/html.

The requested Content-Type is: text/plain; charset=utf-8

jeluf wrote:

Grmpf. Cut&Paste-Error, please excuse.

Should be fixed now.