Page MenuHomePhabricator

VisualEditor: Get rid of ve.splitClusters
Closed, ResolvedPublic

Description

It has a comment that says TODO: strip out calls to splitClusters then delete this method.


Version: unspecified
Severity: enhancement

Details

Reference
bz57628

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:22 AM
bzimport set Reference to bz57628.
Catrope created this task.Nov 26 2013, 10:59 PM

sucheta.ghoshal wrote:

Well, if we are going beyond BMP, we need to take surrogate pairs into account, too. JavaScript does not do that natively, AFAIK. Would code points be enough in that case?

(In reply to Sucheta Ghoshal from comment #1)

Well, if we are going beyond BMP, we need to take surrogate pairs into
account, too. JavaScript does not do that natively, AFAIK. Would code points
be enough in that case?

I think the idea is that at the DM layer, we won't be combining anything beyond code points, but David (CC) can confirm.

dchan added a comment.Oct 7 2014, 3:57 PM

Hi, yes, we wrote ve.splitClusters because we wanted to make the document model be a list of grapheme clusters, instead of a list of raw Javascript characters (i.e. Unicode code units, so each surrogate pair like '\uD860\uDEE2' is treated as two separate entities '\uD860' and '\uDEE2').

However, we've subsequently decided against that, because browsers will not always agree what constitutes a grapheme cluster. The example of Malayalam, where the font can affect the number of clusters, is one example of how problematic it could be to try to match the browser's clusterings exactly.

Therefore, the DM is to remain a list of raw Javascript characters, and support related to clustering is being developed at a level on top of the DM.

Change 165433 had a related patch set uploaded by SuchetaG:
Getting rid of ve.splitClusters in VE core

https://gerrit.wikimedia.org/r/165433

Change 165430 had a related patch set uploaded by SuchetaG:
Getting rid of ve.splitClusters in ve-mw

https://gerrit.wikimedia.org/r/165430

Change 165430 merged by jenkins-bot:
Getting rid of ve.splitClusters in ve-mw

https://gerrit.wikimedia.org/r/165430

Change 165433 merged by jenkins-bot:
Getting rid of ve.splitClusters in VE core

https://gerrit.wikimedia.org/r/165433