Change #1281489 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@master] WIP: Optimize language name loading with fallbacks

https://gerrit.wikimedia.org/r/1281489

Plan as of the end of Wikimedia-Hackathon-2026 (last discussed with @Reedy; also many thanks to @bvibber and @jijiki for insightful discussions):

Given that this Gerrit patch shows counterintuitive performance behavior locally (it’s supposed to be an optimization and locally makes things rather worse), I would like to test the performance of different versions of this in production. (“The performance” will probably be something like “how fast does this API URL return results”; is there something better than time for i in {1..10}; do curl "$url"; done? I should look at Measure backend performance.)

However, testing these changes on mw-experimental doesn’t seem to be feasible (due to the need to rebuild the localisation cache), and Patch Demo or Beta are unlikely to be close enough to production to tell us much about performance. So instead I intend to merge these patches and then backport them; during each backport, I’ll check the performance on mwdebug, and if it gets badly worse, abort the deployment. (It probably makes sense to schedule a dedicated window for this.)

The patches will be (I haven’t fully split them up yet) (updated 7 July 2026):

• Set up the separate i18n directory for language names (added 7 July 2026). Can be merged straight away, as it just updates the migration script that creates the next patch:
• Add CldrMain+LocalNames to i18n files, but separate i18n files (patch to be updated). Translatewiki.net should not offer these for translation yet; we want to investigate the performance first. Expected performance impact: none at all.
• Configure the separate i18n files in MessagesDirs. (Patch TBD.) Expected performance impact: minimal; this will slightly grow the l10n cache, but not by much (some 4.6 MB / 4.4 MiB).
• Load language names from JSON instead of PHP, specifically by reading the JSON files straight from disk. The performance impact of this will be very interesting. (Also roll Update rebuild.php to also write message JSON files into this deploy.)
• Optimize language name loading with fallbacks. This is the other change whose performance impact will be very interesting. Better or worse than reading the JSON files from disk? Better or worse than the previous PHP files?
• (The rest of the chain is cleanup that doesn’t need performance testing.)

If “Load language names from JSON instead of PHP” proves to be significantly slower in production, my plan would be to abort the deploy, but not revert the commit yet; then I would backport “Optimize language name loading with fallbacks” and see if that helps or not. (But at the end of the deployment window, everything should be either merged and deployed, or rolled back and reverted.) If everything works out, then translatewiki.net can configure the new i18n directory to actually let people translate the message names.

Also, I think at this point we’ve decided that this mechanism will not support the MediaWiki: namespace (we’re using either the JSON files or LocalisationCache, but not MessageCache which adds those DB lookups). So while the language names will be i18n messages, it won’t be possible to override them in the MediaWiki: namespace (it’ll just have no effect).

i18n messages

Strictly they would be in i18n banana json format, but they are not "messages". A parallel would be T220759: Provide an alternative to wgExtensionMessagesFiles for non-message i18n and subtasks.

You're probably aware, but noting that there might be some interaction with T99740: Use static php array files for l10n cache at WMF (instead of CDB) with regards to performance.

Change #1308042 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@master] Use separate i18n/LanguageNames subdirectory

https://gerrit.wikimedia.org/r/1308042

In T231755#11886313, @Nikerabbit wrote:

You're probably aware, but noting that there might be some interaction with T99740: Use static php array files for l10n cache at WMF (instead of CDB) with regards to performance.

Indeed, I’m not sure what the best interaction mode is… should we wait for that change to “settle” before finishing this one? Or should we deploy this change now, with the performance testing against the current l10n cache implementation?

Change #1308044 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@master] Add i18n/LanguageNames to MessagesDirs

https://gerrit.wikimedia.org/r/1308044

I think we could proceed as is. No idea how long it might take for the other task to complete.

Change #1308042 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Use separate i18n/LanguageNames subdirectory

https://gerrit.wikimedia.org/r/1308042

Change #1251858 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Add CldrMain+LocalNames to i18n/LanguageNames files

https://gerrit.wikimedia.org/r/1251858

In T231755#12105685, @gerritbot wrote:

Change #1251858 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Add CldrMain+LocalNames to i18n/LanguageNames files

https://gerrit.wikimedia.org/r/1251858

The pashto translation is incorrect, all of them are copy past from Persian, I don't know who did it, I hope it well editable latter in wiki translate.

In T231755#12107482, @ShahZamanPathan wrote:

In T231755#12105685, @gerritbot wrote:

Change #1251858 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Add CldrMain+LocalNames to i18n/LanguageNames files

https://gerrit.wikimedia.org/r/1251858

The pashto translation is incorrect, all of them are copy past from Persian, I don't know who did it, I hope it well editable latter in wiki translate.

The translation comes from upstream CLDR database (https://github.com/unicode-org/cldr/blob/main/common/main/ps.xml) so please create a task here to let Wikimedia's CLDR maintainer representive to fix the issue upstream.

In T231755#12107519, @Bugreporter wrote:

In T231755#12107482, @ShahZamanPathan wrote:

In T231755#12105685, @gerritbot wrote:

Change #1251858 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Add CldrMain+LocalNames to i18n/LanguageNames files

https://gerrit.wikimedia.org/r/1251858

The pashto translation is incorrect, all of them are copy past from Persian, I don't know who did it, I hope it well editable latter in wiki translate.

The translation comes from upstream CLDR database (https://github.com/unicode-org/cldr/blob/main/common/main/ps.xml) so please create a task here to let Wikimedia's CLDR maintainer representive to fix the issue upstream.

Some of them are translated to pashto but some more need to translate, is it possible to edit it after merging

In T231755#12107541, @ShahZamanPathan wrote:

In T231755#12107519, @Bugreporter wrote:

In T231755#12107482, @ShahZamanPathan wrote:

In T231755#12105685, @gerritbot wrote:

Change #1251858 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Add CldrMain+LocalNames to i18n/LanguageNames files

https://gerrit.wikimedia.org/r/1251858

The pashto translation is incorrect, all of them are copy past from Persian, I don't know who did it, I hope it well editable latter in wiki translate.

The translation comes from upstream CLDR database (https://github.com/unicode-org/cldr/blob/main/common/main/ps.xml) so please create a task here to let Wikimedia's CLDR maintainer representive to fix the issue upstream.

Some of them are translated to pashto but some more need to translate, is it possible to edit it after merging

This is not tracked in this task; please file a new task.

In T231755#12105432, @Nikerabbit wrote:

I think we could proceed as is. No idea how long it might take for the other task to complete.

Fair enough. I’ll away for a bit soon (E1999), but can probably try to roll this out the week of July 27–31. I’ll probably try to block a two-hour deployment window for this, given the number of backports (with slow CI; and one of them requiring a full l10n cache rebuild) involved. (See T231755#11884157 for the deployment plan.)

Deployment window scheduled for Wednesday 15:00–17:00 UTC.

Hm, one thing I hadn’t thought about yet is where to deploy this – which train branch do we target, and which wiki do we test the performance on?

Let’s say we backport to wmf.13 (T430832), and test on Test Wikipedia. I expect the testing will consist of ApacheBench (ab), probably like this:

ab -n 1000 -c 24 -l -H 'Host: test.wikipedia.org' 'https://mwdebug.discovery.wmnet:4444/w/api.php?action=query&meta=languageinfo&liprop=name&uselang=de-at&format=json&formatversion=2'

So I’ll create the backport changes tomorrow, once the wmf.13 branch exists. (And if the tests on wmf.13 are successful, then the original changes can be merged on master.)

Change #1318704 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Add i18n/LanguageNames to MessagesDirs

https://gerrit.wikimedia.org/r/1318704

Change #1318705 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Load language names from JSON instead of PHP

https://gerrit.wikimedia.org/r/1318705

Change #1318706 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Update rebuild.php to also write message JSON files

https://gerrit.wikimedia.org/r/1318706

Change #1318707 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Optimize language name loading with fallbacks

https://gerrit.wikimedia.org/r/1318707

Change #1318708 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Remove most of LocalNames

https://gerrit.wikimedia.org/r/1318708

Change #1318709 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Remove T231755.php again

https://gerrit.wikimedia.org/r/1318709

Change #1318704 merged by jenkins-bot:

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Add i18n/LanguageNames to MessagesDirs

https://gerrit.wikimedia.org/r/1318704

Mentioned in SAL (#wikimedia-operations) [2026-07-29T15:14:46Z] <lucaswerkmeister-wmde@deploy1003> Started scap sync-world: Backport for [[gerrit:1318704|Add i18n/LanguageNames to MessagesDirs (T231755)]]

Mentioned in SAL (#wikimedia-operations) [2026-07-29T15:32:52Z] <lucaswerkmeister-wmde@deploy1003> lucaswerkmeister-wmde: Backport for [[gerrit:1318704|Add i18n/LanguageNames to MessagesDirs (T231755)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-07-29T15:46:45Z] <lucaswerkmeister-wmde@deploy1003> Finished scap sync-world: Backport for [[gerrit:1318704|Add i18n/LanguageNames to MessagesDirs (T231755)]] (duration: 31m 59s)

Change #1318705 merged by jenkins-bot:

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Load language names from JSON instead of PHP

https://gerrit.wikimedia.org/r/1318705

Change #1318706 merged by jenkins-bot:

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Update rebuild.php to also write message JSON files

https://gerrit.wikimedia.org/r/1318706

Mentioned in SAL (#wikimedia-operations) [2026-07-29T16:01:12Z] <lucaswerkmeister-wmde@deploy1003> Started scap sync-world: Backport for [[gerrit:1318705|Load language names from JSON instead of PHP (T231755)]], [[gerrit:1318706|Update rebuild.php to also write message JSON files (T231755)]]

Mentioned in SAL (#wikimedia-operations) [2026-07-29T16:07:26Z] <lucaswerkmeister-wmde@deploy1003> lucaswerkmeister-wmde: Backport for [[gerrit:1318705|Load language names from JSON instead of PHP (T231755)]], [[gerrit:1318706|Update rebuild.php to also write message JSON files (T231755)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-07-29T16:16:48Z] <lucaswerkmeister-wmde@deploy1003> Finished scap sync-world: Backport for [[gerrit:1318705|Load language names from JSON instead of PHP (T231755)]], [[gerrit:1318706|Update rebuild.php to also write message JSON files (T231755)]] (duration: 15m 36s)

Change #1318707 merged by jenkins-bot:

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Optimize language name loading with fallbacks

https://gerrit.wikimedia.org/r/1318707

Mentioned in SAL (#wikimedia-operations) [2026-07-29T16:26:05Z] <lucaswerkmeister-wmde@deploy1003> Started scap sync-world: Backport for [[gerrit:1318707|Optimize language name loading with fallbacks (T231755)]]

Mentioned in SAL (#wikimedia-operations) [2026-07-29T16:28:06Z] <lucaswerkmeister-wmde@deploy1003> lucaswerkmeister-wmde: Backport for [[gerrit:1318707|Optimize language name loading with fallbacks (T231755)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Change #1319131 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Revert "Optimize language name loading with fallbacks"

https://gerrit.wikimedia.org/r/1319131

Change #1318708 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Remove most of LocalNames

Reason:

not needed on wmf.13

https://gerrit.wikimedia.org/r/1318708

Change #1318709 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Remove T231755.php again

Reason:

not needed on wmf.13

https://gerrit.wikimedia.org/r/1318709

Change #1319131 merged by jenkins-bot:

[mediawiki/extensions/cldr@wmf/1.47.0-wmf.13] Revert "Optimize language name loading with fallbacks"

https://gerrit.wikimedia.org/r/1319131

Mentioned in SAL (#wikimedia-operations) [2026-07-29T17:02:30Z] <lucaswerkmeister-wmde@deploy1003> Finished scap sync-world: Backport for [[gerrit:1318707|Optimize language name loading with fallbacks (T231755)]] (duration: 36m 25s)

The deployment window is done and I have mostly good news! I documented the results (with mw-collapsible ab outputs) on wiki. Relative to the plan from T231755#11884157:

• Add i18n/LanguageNames to MessagesDirs had essentially no performance impact, as expected. The somewhat increased l10n cache size apparently didn’t make the API slower to respond.
• Load language names from JSON instead of PHP, deployed together with Update rebuild.php to also write message JSON files, also had essentially no performance impact. This means we can go ahead with this work overall \o/
• However, Optimize language name loading with fallbacks made performance significantly more erratic, with much higher standard deviation reported by ab. While the one run with the lowest standard deviation (still much higher than for the other deploys) had the lowest median time of all the test runs I did (i.e., seemingly better than the other deploys), most other runs had much higher median times reported. While I find this difficult to interpret, my conclusion is that this “optimization” is not certainly an improvement and may well be a deterioration (though it’s also possible that mwdebug performance characteristics just changed for other reasons). For now, I reverted this change on wmf.13.
• The rest of the changes aren’t actually needed on wmf.13 and I abandoned them.

For the master branch, I think this means we can merge all the changes except for the “optimize” change (which I’ve just rebased out of the rest of the chain). This will give us translatable language names going forward. Later, once T99740 is done, I want to try the “optimization” again and see if it works better with the new l10n cache implementation.

There's a claim here that a number of patches related to this task should be merged into main: https://phabricator.wikimedia.org/T430833#12176845 for the wmf.14 train. Is there someone here who can provide +2? Are those changes indeed a blocker?

Reading through the comment history it seems wmf.13 was used to run some performance checks on the patches in question, but that the patches themselves are not a blocker for wmf.14.

Tomorrow I will proceed with wmf.14 if the patches have not been merged by then unless there are any objections. Pinging @Lucas_Werkmeister_WMDE and @Bugreporter

It’s not a blocker, but I was hoping someone™ would have merged the patches, because now they were deployed for a few days and then undeployed again which is generally not ideal ^^

But yeah, still waiting for CR+2 from some other CLDR developer :)

Change #1308044 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Add i18n/LanguageNames to MessagesDirs

https://gerrit.wikimedia.org/r/1308044

Change #1329516 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/cldr@master] Add a few more LanguageNames to i18n JSON files

https://gerrit.wikimedia.org/r/1329516

Change #1329620 had a related patch set uploaded (by Raimond Spekking; author: Raimond Spekking):

[translatewiki@master] [cldr] Register LanguageNames directory

https://gerrit.wikimedia.org/r/1329620

Change #1251859 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Load language names from JSON instead of PHP

https://gerrit.wikimedia.org/r/1251859

Change #1251860 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Update rebuild.php to also write message JSON files

https://gerrit.wikimedia.org/r/1251860

Change #1329516 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Add a few more LanguageNames to i18n JSON files

https://gerrit.wikimedia.org/r/1329516

Change #1333852 had a related patch set uploaded (by Siebrand; author: Siebrand):

[mediawiki/extensions/cldr@master] Update inconsistencies in JSON

https://gerrit.wikimedia.org/r/1333852

Change #1333852 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Update inconsistencies in JSON

https://gerrit.wikimedia.org/r/1333852

Change #1329620 merged by jenkins-bot:

[translatewiki@master] [cldr] Register LanguageNames directory

https://gerrit.wikimedia.org/r/1329620

@siebrand: The cleanup task are not yet done (see https://gerrit.wikimedia.org/r/c/mediawiki/extensions/cldr/+/1251861 and https://gerrit.wikimedia.org/r/c/mediawiki/extensions/cldr/+/1251862). And I will prefer this be closed only when we prove translatewiki localization are synced to extension and can be used in Wikimedia projects (currently they are not yet imported to translatewiki, let alone be translated by someone).

Before LocalNames are removed people may try to make patches to update them (e.g. adding a new language), which will not work.

https://translatewiki.net/wiki/FAQ#Language_names and https://translatewiki.net/wiki/CLDR also needs updating. (I will update the MediaWiki.org documentation soon, but I do not have an account in translatewiki.)

Change #1251861 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Remove most of LocalNames

https://gerrit.wikimedia.org/r/1251861

Change #1251862 merged by jenkins-bot:

[mediawiki/extensions/cldr@master] Remove T231755.php again

https://gerrit.wikimedia.org/r/1251862

I’d also like https://translatewiki.net/wiki/Template:Doc-cldr-language-name to be expanded. Reading through the task, I think translations should be made both to translatewiki.net and CLDR, but I’m not entirely sure. Can someone more knowledgeable (e.g. @Lucas_Werkmeister_WMDE, who created the template) expand it to state

• whether translators should submit translations to CLDR as well,
• if yes, how they can do so;
• and also a note that these translations cannot be overridden on-wiki (if they really aren’t).

Changes of CLDR would be regularly imported to CLDR extension. For upstreaming translations to CLDR, see T382979: Workflow to upstream local language name translations in CLDR extension to upstream CLDR project.

I’d also like https://translatewiki.net/wiki/Template:Doc-cldr-language-name to be expanded.

This can be edited in translatewiki without a patch.

and also a note that these translations cannot be overridden on-wiki (if they really aren’t).

They can not be changed on-wiki since the extension reads JSON files directly. See T355496: Allow override language name translations instead of only autonyms in system messages or config options for overriding it.

In T231755#12292581, @Bugreporter wrote:

This can be edited in translatewiki without a patch.

I know. However, I cannot edit it if I don’t know what to write in it.

In T231755#12292581, @Bugreporter wrote:

Changes of CLDR would be regularly imported to CLDR extension. For upstreaming translations to CLDR, see T382979: Workflow to upstream local language name translations in CLDR extension to upstream CLDR project.

So people should just submit translations on translatewiki.net only and hope that the upstreaming development is ready before the next import from CLDR (so that their translations are not overwritten from CLDR)?

If the CLDR translation is wrong, we can fix it upstream using Wikimedia's voting right (though the survey tool only opens for several months each year); individual locales can also be added in same way. There are currently no process to batch upstream translations that are not yet in CLDR.

If a language name in JSON is different from the CLDR one, it will not be overwritten (this is already the case previously, when local overrides are in LocalNames).

There are no i18n bot activity in https://phabricator.wikimedia.org/diffusion/ECLD/history/master/. Can someone comfirm export works properly?

Too hasty. Patch was merged on Friday (and Friday exports had a failure due to SimilarEdits) and there are no exports during weekend.

First export from translatewiki now on Monday morning: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/cldr/+/1337539

In T231755#12292646, @Tacsipacsi wrote:

So people should just submit translations on translatewiki.net only and hope that the upstreaming development is ready before the next import from CLDR (so that their translations are not overwritten from CLDR)?

In T231755#12292650, @Bugreporter wrote:

If a language name in JSON is different from the CLDR one, it will not be overwritten (this is already the case previously, when local overrides are in LocalNames).

Correct. There’s a TODO in the code that it would be nice to mark the translation as fuzzy in that case, so translatewiki.net volunteers can go check if the override should be adjusted, but that’s not implemented at the moment.