Tamil sort order
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• bzimport
	Nov 15 2014, 8:56 AM

Description

Author: sundarbecse

Description:
Showing Tamil consonant sequences

Reference page: https://ta.wikipedia.org/s/4om
If you see in the above page, 'ஜ' follows 'ச'. Characters like 'ஸ', 'ஷ', 'ஜ', 'ஹ' etc. are called grantha characters which are not part of the basic alphabets of Tamil. See https://en.wikipedia.org/wiki/Tamil_script#Basic_consonants They are added towards the end (i.e. after 'ன') by convention. The first column in the attached image shows the correct sequence. (Image source: Naga. Ilangovan)

Version: unspecified
Severity: enhancement

Attached:

alphabetsequence.jpg (494×328 px, 29 KB)

To fix this, we'll need to change the collation to uca-ta collation, and run the updateCollation.php script for all the Tamil wikis.

Details

Reference: bz73453

Subject	Repo	Branch	Lines +/-
Revert "Revert "Set Tamil projects to use uca-ta collation""	operations/mediawiki-config	master	+6 -0
Add support for icu-ta collation	mediawiki/core	master	+5 -0
Adding Tamil to IcuCollation::$tailoringFirstLetters	mediawiki/core	master	+1 -0
Set Tamil projects to use uca-ta collation	operations/mediawiki-config	master	+6 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T32996 Change $wgCategoryCollation values to appropriate one for each Wikimedia wiki
Open	Feature	None	T47443 Deploy language-specific "uca-xx" collations on Wikimedia wikis
Resolved		kaldari	T75453 Tamil sort order

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 4:00 AM

• bzimport added a project: MediaWiki-Internationalization.

• bzimport set Reference to bz73453.

• bzimport added a subscriber: Unknown Object (MLST).

• bzimport created this task.Nov 15 2014, 8:56 AM

I think we need to add Collation support for Tamil (not sure if we need upstream libicu stuff), and look at getting the category collation updated on tawiki

ICU appears to support Tamil (http://bugs.icu-project.org/trac/browser/icu/trunk/source/data/coll/ta.txt), so we only need to add it to the list of supported collations and perhaps adjust first-letter generation. (And then confirm that it actually sorts the words correctly.)

sundarbecse wrote:

Thanks Same Reed and Bartosz Dziewoński. Yes, http://bugs.icu-project.org/trac/browser/icu/trunk/source/data/coll/ta.txt is correct for the consonant sequence. We just need to validate the overall sequence of vowels, consonants, compounds.

nelango5 wrote:

Following are the two other related issues that I would like to be added to this bug report.

The sort position of letter ஃ should be after all the vowels. Currently, it is positioned like ஃ, அ, ஆ, இ, ஈ, உ, ஊ, எ, ஏ, ஐ, ஒ, ஓ, ஔ. The right order is அ, ஆ, இ, ஈ, உ, ஊ, எ, ஏ, ஐ, ஒ, ஓ, ஔ, ஃ. It should be noted that, upon sorting all the Tamil letters, ஃ should appear after ஔ and before க்.

The Consonant letters should appear on top of their compounding forms. If we sort the letters (ம, ம், மா ), the right result is ( ம், ம, மா ). Currently the order is (ம, மா, ம் ) which is wrong. The impact of this can be understood by sorting a few strings. Given the set of 4 strings as (கணமொழி, கணமூலி, கணம்புல், கணம்), current sort order results into (கணமூலி, கணமொழி, கணம், கணம்புல்). This is wrong and the right order is (கணம், கணம்புல், கணமூலி, கணமொழி). (These 4 strings are proper Tamil words according to Tamil lexicon @ http://dsalsrv02.uchicago.edu/dictionaries/tamil-lex/ .

sundarbecse wrote:

(In reply to elan from comment #4)

Following are the two other related issues that I would like to be added to
this bug report.

The sort position of letter ஃ should be after all the vowels. Currently,

it is positioned like ஃ, அ, ஆ, இ, ஈ, உ, ஊ, எ, ஏ, ஐ, ஒ, ஓ, ஔ. The right
order is அ, ஆ, இ, ஈ, உ, ஊ, எ, ஏ, ஐ, ஒ, ஓ, ஔ, ஃ. It should be noted that,
upon sorting all the Tamil letters, ஃ should appear after ஔ and before க்.

The Consonant letters should appear on top of their compounding forms. If

we sort the letters (ம, ம், மா ), the right result is ( ம், ம, மா ).
Currently the order is (ம, மா, ம் ) which is wrong. The impact of this can
be understood by sorting a few strings. Given the set of 4 strings as
(கணமொழி, கணமூலி, கணம்புல், கணம்), current sort order results into (கணமூலி,
கணமொழி, கணம், கணம்புல்). This is wrong and the right order is (கணம்,
கணம்புல், கணமூலி, கணமொழி). (These 4 strings are proper Tamil words according
to Tamil lexicon @ http://dsalsrv02.uchicago.edu/dictionaries/tamil-lex/ .

Yes, the above sequence is the correct order.

Liuxinyu970226 subscribed.Jan 24 2015, 5:49 AM

Info-farmer subscribed.Jun 24 2015, 9:14 AM

Mahir78 subscribed.Aug 21 2015, 7:14 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 21 2015, 7:14 AM

Logicwiki subscribed.Oct 20 2015, 10:34 AM

kaldari added a project: Community-Tech.May 24 2016, 5:18 PM

kaldari updated the task description. (Show Details)

kaldari moved this task from New & TBD Tickets to Up Next (May 20-June 3) on the Community-Tech board.May 24 2016, 5:29 PM

kaldari removed a project: Community-Tech.

kaldari updated the task description. (Show Details)May 24 2016, 7:28 PM

Change 290529 had a related patch set uploaded (by Kaldari):
Set Tamil projects to use uca-ta collation

https://gerrit.wikimedia.org/r/290529

gerritbot added a project: Patch-For-Review.May 24 2016, 8:24 PM

Change 290529 merged by jenkins-bot:
Set Tamil projects to use uca-ta collation

https://gerrit.wikimedia.org/r/290529

Mentioned in SAL [2016-05-24T23:20:04Z] <dereckson@tin> Synchronized wmf-config/InitialiseSettings.php: Set Tamil projects to use uca-ta collation (T75453) (duration: 02m 18s)

matmarex mentioned this in T86096: Switch HAT appservers to trusty's ICU (or newer).May 25 2016, 1:10 PM

Um, folks, you broke it :( All category pages are unviewable now.

The exception is "MediaWiki does not support ICU locale "ta"", which it indeed doesn't. The 'uca-ta' collation first has to be defined in /includes/collation/IcuCollation.php in MediaWiki core, by adding an entry for 'ta' to the $tailoringFirstLetters variable. I mentioned this here some time ago (T75453#775155).

The exception is "MediaWiki does not support ICU locale "ta"", which it indeed doesn't. The 'uca-ta' collation first has to be defined in /includes/collation/IcuCollation.php in MediaWiki core, by adding an entry for 'ta' to the $tailoringFirstLetters variable. I mentioned this here some time ago (T75453#775155).

Also as a reminder for next time, immediately after changing the collation settings, you have to run the updateCollation.php script, or things will be broken.

(This was reverted with https://gerrit.wikimedia.org/r/290682 and deployed now.)

matmarex removed a project: Patch-For-Review.May 25 2016, 1:32 PM

matmarex set Security to None.

matmarex removed a subscriber: • wikibugs-l-list.

Change 290685 had a related patch set uploaded (by Dereckson):
Add support for icu-ta collation

https://gerrit.wikimedia.org/r/290685

gerritbot added a project: Patch-For-Review.May 25 2016, 1:41 PM

Change 290686 had a related patch set uploaded (by Dereckson):
Revert "Revert "Set Tamil projects to use uca-ta collation""

https://gerrit.wikimedia.org/r/290686

kaldari closed this task as Resolved.May 27 2016, 11:34 PM

kaldari claimed this task.

Oops, looks like this isn't actually resolved.

Change 291380 had a related patch set uploaded (by Kaldari):
Adding Tamil to IcuCollation::$tailoringFirstLetters

https://gerrit.wikimedia.org/r/291380

Change 291380 abandoned by Kaldari:
Adding Tamil to IcuCollation::$tailoringFirstLetters

Reason:
Redundant to https://gerrit.wikimedia.org/r/#/c/290685/

https://gerrit.wikimedia.org/r/291380

Change 290685 merged by jenkins-bot:
Add support for icu-ta collation

https://gerrit.wikimedia.org/r/290685

ReleaseTaggerBot added projects: MW-1.28-release-notes, MW-1.28-release (WMF-deploy-2016-06-07_(1.28.0-wmf.5)).Jun 1 2016, 2:00 AM

Dereckson added a project: Wikimedia-Site-requests.Jun 9 2016, 7:30 PM

Restricted Application added subscribers: JEumerus, Matanya. · View Herald TranscriptJun 9 2016, 7:30 PM

Dereckson moved this task from Backlog to To deploy on the Wikimedia-Site-requests board.Jun 9 2016, 7:31 PM

Change 290686 merged by jenkins-bot:
Revert "Revert "Set Tamil projects to use uca-ta collation""

https://gerrit.wikimedia.org/r/290686

Mentioned in SAL [2016-06-09T23:57:31Z] <dereckson@tin> Synchronized wmf-config/InitialiseSettings.php: Set Tamil projects to use uca-ta collation II (T75453) (duration: 00m 25s)

23:57 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set Tamil projects to use uca-ta collation II (T75453) (duration: 00m 25s)
00:07 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawiktionary --force
00:12 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikisource --force
00:31 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikiquote --force
00:39 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikinews --force
00:40 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikibooks --force
00:56 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawiki --force

Liuxinyu970226 unsubscribed.Jun 10 2016, 1:18 AM

Nemo_bis edited projects, added Wikimedia-Language-setup; removed Wikimedia-Site-requests, MediaWiki-Internationalization.Jun 10 2016, 7:20 AM

Nemo_bis added a project: MediaWiki-Internationalization.

MarcoAurelio added a project: Tamil-Sites.Dec 22 2016, 12:31 PM

MarcoAurelio removed a parent task: T34578: [DO NOT USE] Tamil Wiki projects (tracking) [superseded by #Tamil-Sites].Dec 22 2016, 12:34 PM