CLDR in MediaWiki
Open, MediumPublic
Actions

Assigned To

None

Authored By

	JnpoJuwan
	Jan 3 2025, 11:15 PM

Description

I wish to request a new method that we implement CLDR on MediaWiki, as it currently leaves a lot to be desired regarding adding and modifying content.

I started investigating this as I am working on translating strings to the language Toki Pona (tok). In preparation for the possible Toki Pona Wikipedia and for adding support for existing Toki Pona wikis, me and other contributors have been translating strings and have come across the fact that there are several missing and simply incorrect pieces of information relating to the language locale. For example, 'Bulgarian' (bg) is translated as 'ma Pokasi' meaning 'Bulgaria' (the country); and the date formats have not been properly marked: 3 January 2025 is '3 tenpo mun #1 2024', however speakers would rather say 'sike #2025 la mun #1 la suno #3' or similar. In order to edit Toki Pona items, I contacted CLDR to create a Survey Tool account. However, the project only accepts updates for a short period of time, requires a lot of votes to pass and takes months to be released.

Below are a few other related tasks and discussions, where people had to endure with incorrect or missing information for a long time (some tasks are still even open), brought to my attention from the kind people at the Wikimedia community Discord server. This is not a problem just for the small languages but for all languages.

Previous discussions

This is an issue because as far as I am aware, contributors choose to stick to CLDR and not interfere with it locally, as one pointed out to edit CLDR directly rather than MediaWiki. With the difficulty of editing, however, this is not possible for most contributors. With the recent membership of the Wikimedia Foundation to Unicode, I suggest that it help contributors to add local strings upstream to the CLDR.

Stjn proposed a basic workflow which I will reiterate here as a proposal:

Post information regarding the desired changes to Village Pump of relevant language Wikipedia (possibly other projects as well)
Update local CLDR extension files if approved by contributors
A WMF member raises the issue upstream and votes for it being fixed

This would be better as a method to more quickly update errors on our projects and more quickly update CLDR.
I hope to hear feedback on this idea.

Related Objects

Mentioned In: T364334: Unicode CLDR should be the fallback data instead of the "true and only source"
Mentioned Here: T151269: Add English names for languages which don't yet have one
T168799: Integrate IANA language registry with language-data and MediaWiki (let MediaWiki "knows" all languages with ISO 639-1/2/3 codes)
T231755: Local language name should be translatable in translatewiki.net
T364334: Unicode CLDR should be the fallback data instead of the "true and only source"
T189511: Locally override the name of crh from "Crimean Turkish" to "Crimean Tatar"
T231748: language names in Dutch missing at Wikidata
T245927: Wrong Russian translation for ISO 639 language code "mul" (add mul - "несколько языков" to LocalNamesRu)
T256649: incorrect English names for languages (they display the native names only)

Event Timeline

JnpoJuwan created this task.Jan 3 2025, 11:15 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2025, 11:15 PM

Reedy added a project: LPL Onboarding and Development.Jan 3 2025, 11:18 PM

requires a lot of votes to pass

The CLDR concept of "votes" is misleadingly named. The Wikimedia Foundation is a Unicode organization member so accounts set up by @Nemo_bis following the process at https://translatewiki.net/wiki/CLDR are by themselves capable of providing 4 votes, which is enough to change something for the kinds of low-use languages typically involved in this process like Toki Pona by themselves.

(the rest of your criticism of CLDR is definitely valid, just felt that one issue deserved correction)

This looks very much like a duplicate of T364334

If you want to upstream data to CLDR, there are points to considered:

There are several language codes that is local to MediaWiki, e.g. zh-classical. Such codes must be filter out and never be upstreamed.
Once T231755: Local language name should be translatable in translatewiki.net is resolved, anyone in translatewiki can translate the language name of any languages
- But if we automatically upstream them, there will be potential risk for vandalism.
- In addition, some language names are overrided locally in MediaWiki (e.g. T231755#9766845) and I am not sure whether we want to upstream them.
New languages are added to CLDR extension for various usages (such as monolingual text language in Wikidata, or someone want to translate it in translatewiki). However previously CLDR have stated it is not intended to cover every languages, see T151269#2822033 (I am not sure whether it is still the case today).
- In extreme cases, T168799: Integrate IANA language registry with language-data and MediaWiki (let MediaWiki "knows" all languages with ISO 639-1/2/3 codes) proposed to add 7000 language codes to CLDR extension.

Bugreporter mentioned this in T364334: Unicode CLDR should be the fallback data instead of the "true and only source".Jan 4 2025, 12:18 AM

@Bugreporter, I don't believe that it would be helpful to upstream these automatically. I can definitely see the danger there. this would ideally firstly need to gather approval from multiple speakers before being pushed.

@Pppery, thank you for the correction and for pointing out the other tasks. I have subscribed to these and will keep a look for updates.

this would ideally firstly need to gather approval from multiple speakers before being pushed.

Yeah, but once you can translate local language names in translatewiki, the number of localizations that may be upstreamed may be too many to be checked one-by-one. In additional in several languages there are few but still some translators - which means "gather approval from multiple speakers" may be impossible.

there would still need to be some confirmation from other speakers, may them be locally in Wikimedia or in CLDR:

any user may add a new translation for MediaWiki
other users may review it locally and that translation may be passed upstream to CLDR
Wikimedia may add a translation to CLDR as unconfirmed
with confirmation from other translators on CLDR, it may be voted upon and confirmed (with the help of votes from Wikimedia)

Nemoralis subscribed.Jan 20 2025, 10:08 PM

SToyofuku-WMF subscribed.Feb 28 2025, 9:40 PM

MaryMunyoki moved this task from Backlog to Incoming Requests on the LPL Onboarding and Development board.Mar 11 2025, 3:50 PM

Aklapper merged a task: T364334: Unicode CLDR should be the fallback data instead of the "true and only source".Mar 20 2025, 9:03 AM

Aklapper added subscribers: Winston_Sung, srishakatux, Nikerabbit, Amire80.

MaryMunyoki moved this task from Incoming Requests to Backlog on the LPL Onboarding and Development board.Tue, Apr 1, 7:39 PM

MaryMunyoki triaged this task as Medium priority.Thu, Apr 10, 4:42 PM

MaryMunyoki moved this task from Backlog to Quarter Backlog on the LPL Onboarding and Development board.