MediaWiki fails to parse  (en dash)
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	Tgr
	Sep 20 2015, 9:34 PM

Description

Steps to reproduce: enter  in the wikitext of the page
Expected result: an en dash (–) is displayed.
Actual result:  is displayed (on the HTML level, & gets encoded into &.

The relevant spec is consume a character reference in HTML5, which has a compatibility table for frequently used character codes which are not Unicode code points. (En dash is U+2013 so the straightforward representation is – but the HTML5 standard also acknowledges Windows-1250 codes for a number of characters which when interpreted as Unicode would result in unprintable characters - U-0096 is "start of guarded area".) MediaWiki should respect that compatibility table when encoding references.

Details

	Subject	Repo	Branch	Lines +/-
	Ensure PHP and JS are consistent wrt allowed entities	mediawiki/services/parsoid	master	+98 -35

Customize query in gerrit

Related Objects

Mentioned In: T219943: Create a composer library for wikipeg
T221384: Wikipeg cache is unsafe when rule variables are set to null/undefined
T219938: Port HTML5Treebuilder
T219072: Extend JS/PHP hybrid testing to other Parsoid components
T205338: Extract and use a DOMHandler interface (API) for HTML -> wt handlers defined on DOM nodes
T106578: Sanitizer entity restrictions should match HTML5
Mentioned Here: rGPARc9dab9d47fde: Update wikipeg to 2.0.3 (PHP side)
T205338: Extract and use a DOMHandler interface (API) for HTML -> wt handlers defined on DOM nodes
T219072: Extend JS/PHP hybrid testing to other Parsoid components
T219938: Port HTML5Treebuilder
T219943: Create a composer library for wikipeg
T221384: Wikipeg cache is unsafe when rule variables are set to null/undefined
T106578: Sanitizer entity restrictions should match HTML5

Event Timeline

Tgr created this task.Sep 20 2015, 9:34 PM

Tgr raised the priority of this task from to Needs Triage.

Tgr updated the task description. (Show Details)

Tgr added a project: MediaWiki-Parser.

Tgr subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 20 2015, 9:34 PM

This behaviour is obviously caused by T106578.

It is implemented by function validateCodepoint within rMW/includes/Sanitizer.php

My personal view: Undefined Unicode values, either raw or as entity, should be removed from wiki project source texts and not being supported and encouraged in eternity. German WP assumes CP1252, others CP1250; the cyrillic author intended something entirely different and it always became visible on Russian WP in local style, but changing its meaning according to this task.

In T113194#1664501, @PerfektesChaos wrote:

German WP assumes CP1252, others CP1250

The HTML5 spec clearly specifies CP1250. There is no reason we have to go with the standard but that requires the least effort.

Our authors in Greece inserted that thing and saw CP1253 the last decade, colleagues in Moscow got CP1251 displayed, someone in Istanbul was happy with presentation as CP1254.

This went on for 15 years now, and our texts have been acquired to this long before HTML 5.1 invented that new behaviour.

German WP is currently eliminating all occurrences; we regard them as CP1252.

The behaviour implemented by Tim is exactly how conversion should happen: An ambiguous entity is presented in the article, leaving it to the authors to replace that by valid UCS. It must not displayed a 1250 Š while the author intended and always saw a Љ (8A/1251).

MediaWiki should not maintain dubios codes; this is not an HTML editor. If bad codes are inserted by c&p, they should attract attention immediately, not silently converted from Vietnamese to east European.

Fair point.

Thank you for withdrawal.

One more aspect: If I am sitting in Minsk and insert a string within source text editing by c&p, I can see a Љ in the edit area as expected. On preview I am looking at Š. Quite spooky. Remains miraculous even when re-editing the page. A  displayed will give an IT freak a pretty good hint.

Greetings

cscott mentioned this in T106578: Sanitizer entity restrictions should match HTML5.Dec 1 2017, 9:27 PM

Change 504621 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/services/parsoid@master] Ensure PHP and JS are consistent wrt allowed entities

https://gerrit.wikimedia.org/r/504621

gerritbot added a project: Patch-For-Review.Apr 17 2019, 5:28 PM

Change 504621 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Ensure PHP and JS are consistent wrt allowed entities

https://gerrit.wikimedia.org/r/504621

Mentioned in SAL (#wikimedia-operations) [2019-04-29T20:40:46Z] <arlolra> Updated Parsoid to c9dab9d (T106578, T113194, T205338, T219072, T219938, T221384, T219943)