The usage of ↑ seems to be new in version 46 in comparison to version 45…

At first glance this seems to be a change in the source data that's not being taken into account by the extension scripts. This needs attention.

Change #1317543 had a related patch set uploaded (by Siebrand; author: Siebrand):

[mediawiki/extensions/cldr@master] Handle CLDR inheritance marker (↑↑↑) in CLDRParser

https://gerrit.wikimedia.org/r/1317543

Change #1317544 had a related patch set uploaded (by Siebrand; author: Siebrand):

[mediawiki/extensions/cldr@master] Rebuild CLDR data with inherited language names

https://gerrit.wikimedia.org/r/1317544

Change #1333882 had a related patch set uploaded (by Siebrand; author: Siebrand):

[mediawiki/extensions/cldr@master] Re-add self-referential language names after parent inheritance

https://gerrit.wikimedia.org/r/1333882

Change #1317543 abandoned by Siebrand:

[mediawiki/extensions/cldr@master] Handle CLDR inheritance marker (↑↑↑) in CLDRParser

Reason:

Replaced by I9911866a59dabcdccc8fcc4f954e745a4e8dc307 because I091a22760fe1082269690c7d2d0ef281edce31b4 is better.

https://gerrit.wikimedia.org/r/1317543

Change #1317544 abandoned by Siebrand:

[mediawiki/extensions/cldr@master] Rebuild CLDR data with inherited language names

Reason:

Parent was abandoned. This is generated data.

https://gerrit.wikimedia.org/r/1317544

The CLDR release files do not provide distinction between missing/fallback values because they assume a normal CLDR inheritance is applied. This is not he case in MediaWiki were we apply language fallbacks. Any heuristics trying to infer this will have false positives.

The only options that I see are:

• Use different data source with inheritance markers
• Apply the CLDR inheritance algorithm before going for MediaWiki fallbacks. I am afraid this would mostly default to language code for codes known in CLDR and prevent using our own local translations.

I991186 was proposed to fix a regression introduced in CLDR 46, where language names identical to their code (fon, luo, tiv, vai, yao) were dropped from locale files. The patch reads en.xml, identifies codes whose English name equals the code, and re-injects them after parent-locale inheritance for any locale still missing them.

@Nikerabbit raised that the CLDR release files carry no inheritance markers, so any heuristic inferring inheritance state will have false positives. @Pikne raised the same concern via a concrete example (gaa in Estonian), and additionally noted that injected names should be title-cased and English-only — the latter being a separate feature request, not addressed here.

The approach in both patches I991186 (initial) and I949119 (reworked) is in my opinion correct and justified: MediaWiki's fallback chain cannot be relied upon to mirror CLDR's parent chain, so generated files must be self-contained. A missing language name in a generated file does not fall back to the CLDR value — it falls back through MediaWiki's language fallback chain, which does not always reach English and which does not replicate CLDR inheritance. The code-as-name entries must therefore be present in every generated file that would otherwise be missing them.

A reworked version of the patch has been prepared that refactors the injection into a dedicated applyCodeAsNameFallback() method on CLDRParser, and adds tests covering both the detection of self-referential codes and the fallback injection itself. The logic and output are functionally identical to the original patch.

@Nikerabbit: Does this address your concern? The heuristic (reading self-referential codes from en.xml) is the same as in the original patch — the argument for it is that these codes have no name anywhere in the CLDR chain, so the code itself is the correct display name, and omitting it would cause MediaWiki to fall back incorrectly through its own chain rather than to the code. Is there a specific false positive case you have in mind that this approach would not handle correctly?

@Pikne: Does this address your concern? The gaa/et example does not produce a false positive here — gaa is not self-referential in English ("Ga" ≠ "gaa") so it would not be injected. Are there cases you can think of where this would produce a wrong result?

Change #1335783 had a related patch set uploaded (by Siebrand; author: Siebrand):

[mediawiki/extensions/cldr@master] Apply LDML code-as-name fallback after parent inheritance

https://gerrit.wikimedia.org/r/1335783

Change #1335784 had a related patch set uploaded (by Siebrand; author: Siebrand):

[mediawiki/extensions/cldr@master] Rebuild CLDR data with LDML code-as-name fallback applied

https://gerrit.wikimedia.org/r/1335784

In T414677#12288779, @siebrand wrote:

@Pikne /.../ additionally noted that injected names should be title-cased and English-only — the latter being a separate feature request, not addressed here.

I think that patch comment, actually by another user, just somewhat misunderstood the purpose of given patch – English names, capitalized and not identical to code, are not missing and don't have to be re-added.

But other than that the comment is right that particular codes should not be added as language names in all languages where these language names are missing, as shown in now-abandoned patch, but only in languages that actually use these lowercase three-letter Latin-script names, i.e. names in languages for which CLDR actually uses inheritance markers and that existed here as well prior to upgrade to CLDR 46. Which I think is about the same what Nikerabbit meant.

@Pikne Does this address your concern? The gaa/et example does not produce a false positive here — gaa is not self-referential in English ("Ga" ≠ "gaa") so it would not be injected. Are there cases you can think of where this would produce a wrong result?

To be honest I don't understand this last proposal well. My point was that 'gaa' should be injected (re-added) in Estonian, not the other way around. To "resolve the full CLDR inheritance chain" does sound like what is needed but then again if data source still is the same one that doesn't use intheritance markers and where inherited names as simply skipped, then... how?

Edit: is the last data rebuild patch what this proposal results in? As far as I can see it has all the same problems that previous rebuild attempt had – code as language name is added in many languages where this definitely is not appropriate, most notably in languages that don't use Latin-script but also in languages where language names are capitalized like in English, and in other cases in which name in another language differs from English name in more aspects, not only by capitalization. E.g. lowercase language name 'luo' was removed (only) in 28 languages with upgrade to CLDR 46 but your patch "re-adds" it in 140 languages.