Page MenuHomePhabricator

Add tr_TR.utf8 locale to ToolForge
Closed, ResolvedPublic

Description

Can we add tr_TR.utf8 locale to ToolForge server?

As a Turkish Wikipedian and a bot owner, I have language problems on server.

Found a similar task. Might be useful T191727: Add mr_IN locale to toolsforge

Currently installed languages:

tools.mavrikant@tools-sgebastion-07:~/pywikibot$ locale -a
bokmal
C
C.UTF-8
da_DK
da_DK.iso88591
da_DK.utf8
danish
dansk
de_DE
de_DE.iso88591
de_DE.utf8
deutsch
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZW.utf8
fa_IR.utf8
fi_FI
fi_FI.iso88591
fi_FI.utf8
finnish
french
fr_FR
fr_FR.iso88591
fr_FR.utf8
german
it_CH.utf8
it_IT.utf8
ko_KR.utf8
mr_IN.utf8
nb_NO
nb_NO.iso88591
nb_NO.utf8
nn_NO
nn_NO.iso88591
nn_NO.utf8
no_NO
no_NO.ISO-8859-1
norwegian
nynorsk
pl_PL
pl_PL.iso88592
pl_PL.utf8
polish
portuguese
POSIX
pt_PT
pt_PT.iso88591
pt_PT.utf8
romanian
ro_RO
ro_RO.iso88592
ro_RO.utf8
ru_RU
ru_RU.iso88595
ru_RU.utf8
sv_SE
sv_SE.iso88591
sv_SE.utf8
swedish
tg_TJ.utf8

Event Timeline

Reedy renamed this task from Add tr_TR.utf8 locale to ToolForge to Add tr_TR.utf8 locale to ToolForge.Sep 20 2020, 1:06 AM
Reedy updated the task description. (Show Details)
Reedy updated the task description. (Show Details)

The referenced task with the language pack installs is referring to how this was done on Ubuntu. Since we are on Debian now, the procedure will be a bit different in terms of puppet changes. We should also check the base docker images used in Kubernetes.

The same locale(s) should also be added to the Kubernetes Docker containers. Unfortunately this is managed in a separate location which makes it easy to forget to update.

Change 628879 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: use locales::all for the grid

https://gerrit.wikimedia.org/r/628879

So, having installed locales-all on a test Debian server, it installs 480 locales instead of our current 76. There is a clear increase in disk space usage that does not equal a GB (but I think it is hundreds of MB). The most obvious change is that our servers become instantly less Eurocentric. Our current default locales on Debian are based on the needs of the listserv and RSS servers, and adding this to them may not be useful.

Just to test exactly what this changes (running apt clean before checking disk space both times):
Without locales-all

Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/vda2       19478140 11888124   6577540  65% /

With locales-all

Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/vda2       19478140 12000844   6464820  65% /

So we are talking about 112,720 KB on disk.

There are no language packs for Debian in the same way as Ubuntu, so if there are specific font needed for support, they have to be named and expressly installed. For the most part, we have a pretty good font collection for most cases, I think.

If we really don't want to expand this by 113 MB (which seems less important on grid servers...possibly more important on k8s, but maybe still good), we may want to look into refactoring the extended locales to be able to accept our own file listing the locales for Toolforge. The only serious win (besides i18n) in k8s is that the image builds wouldn't need to actually generate locales.

Change 628879 merged by Bstorm:
[operations/puppet@production] toolforge: use locales::all for the grid

https://gerrit.wikimedia.org/r/628879

Is this resolved? As I see, this locale has been installed (using locale -a command)?

Change 632582 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/docker-images/toollabs-images@master] locales: switch to using locales-all package

https://gerrit.wikimedia.org/r/632582

Change 632582 merged by jenkins-bot:
[operations/docker-images/toollabs-images@master] locales: switch to using locales-all package

https://gerrit.wikimedia.org/r/632582

Mentioned in SAL (#wikimedia-cloud) [2020-10-08T17:07:46Z] <bstorm> rebuilding docker images with locales-all T263339

Bstorm claimed this task.

Confirmed all images going forward have all locales with webservice shell.