Page MenuHomePhabricator

Toolforge is missing locales (breaks Mosh and causes various other problems)
Closed, ResolvedPublic

Description

I now get this error while trying to connect to dev.toolforge.org with my normal environment:

$ mosh toolforge-dev
The locale requested by LANG=C.UTF-8 isn't available here.
Running `locale-gen C.UTF-8' may be necessary.

The locale requested by LANG=en_US.UTF-8 isn't available here.
Running `locale-gen en_US.UTF-8' may be necessary.

mosh-server needs a UTF-8 native locale to run.

Unfortunately, the local environment (LANG=C.UTF-8) specifies
the character set "US-ASCII",

The client-supplied environment (LANG=en_US.UTF-8) specifies
the character set "US-ASCII".

locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME="en_US.UTF-8"
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
Connection to dev.toolforge.org closed.
/usr/bin/mosh: Did not find mosh server startup message. (Have you installed mosh on your server?)

I can work around it with LC_ALL=C.UTF-8, but I’m sure it used to work with my normal locale settings, and it still works with used to work with login.toolforge.org (before that was changed as well).

Event Timeline

I also get a locale error (“manpath: can't set the locale; make sure $LC_* and $LANG are correct”) when connecting to toolforge-dev with normal SSH:

$ ssh toolforge-dev
Linux tools-bastion-12 6.1.0-17-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64
Debian GNU/Linux 12 (bookworm)
tools-bastion-12 is a Toolforge bastion (wmcs::toolforge::bastion)
======================================================================
_______  _____   _____         _______  _____   ______  ______ _______
   |    |     | |     | |      |______ |     | |_____/ |  ____ |______
   |    |_____| |_____| |_____ |       |_____| |    \_ |_____| |______
======================================================================
This is a server of the tools Cloud VPS project, the home of community
managed bots, webservices, and tools supporting the Wikimedia movement.

Use of this system is subject to the Toolforge Terms of Use,
Code of Conduct, and Privacy Policies:
- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Terms_and_conditions

General guidance and help can be found at:
- https://toolforge.org/

Host has been migrated to puppet7
The last Puppet run was at Tue Apr 16 15:13:40 UTC 2024 (12 minutes ago). 
Last Puppet commit: (c1a9990e1c) Marostegui - db2127: Disable notifications
Last login: Tue Apr 16 15:25:50 2024 from ip5f5aea01.dynamic.kabel-deutschland.de
manpath: can't set the locale; make sure $LC_* and $LANG are correct
lucaswerkmeister-wmde@tools-bastion-12:~$

(And a similar error if I look up man manpath.)

Previous bastions had (due to the grid) all the locales which I'd like to avoid.

I guess that’s what I used to rely on. If it isn’t going to be supported anymore, maybe remove LANG and LC_* from the server’s AcceptEnv?

After today’s DNS change, this affects login.toolforge.org as well; I assume it’s also the reason I’m now seeing broken UTF-8 in my commit messages:

image.png (503×1 px, 102 KB)

(That’s supposed to be U+2019 RIGHT SINGLE QUOTATION MARK.)

LucasWerkmeister renamed this task from Cannot connect to dev.toolforge.org using Mosh with custom locale to Toolforge is missing locales (breaks Mosh and causes various other problems).Wed, Apr 17, 6:25 PM
LucasWerkmeister updated the task description. (Show Details)

@taavi since you mentioned cultural imperialism in that commit message… this is pretty meh :/

image.png (322×1 px, 56 KB)

The main thing I'm confused here is that why are the locales not breaking on any other instance. The previous bastions having all the locales installed was due to a specific configuration to do that, not the default for all instances. Does this happen on, say, tools-mail-4.tools.eqiad1.wikimedia.cloud, which also has NFS home directories mounted but no special locale config?

It happens there as well, yes.

image.png (487×1 px, 105 KB)

It’s probably not new in general, but it sucks that it now affects the Toolforge bastions, and the Mosh problem in particular wouldn’t have been possible to observe on other cloud VPS instances anyway (since they’re incompatible with Mosh in general for network reasons).

Ok, thanks. After giving this a bit more thought:

  • This really should work out of the box. I feel like something very cursed is happening on these instances where it does not.
  • https://gerrit.wikimedia.org/r/c/operations/puppet/+/1012615 fixed one symptom of the issue, it didn't cause it.
  • The bastion hosts will by design have many people working interactively on them. So offering more user customization (i.e. all the locales) doesn't feel that wrong, compared to doing that on all the backend hosts.
  • So let's just install locales-all on the new bastions too, which should make the problem go away.
  • For the rest of the servers, if this becomes a larger problem, we can look at other solutions (e.g. clearing the sshd default AcceptEnv config options).

Does that sound good to everyone?

Sounds good to me; though I’m also curious what other “shared SSH”-style environments do (e.g. if the AcceptEnv adjustment is common or not). Sadly I don’t have access to a university network anymore that I could compare to 😔

Change #1020906 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::bastion: install locales-all

https://gerrit.wikimedia.org/r/1020906

Change #1020906 merged by Majavah:

[operations/puppet@production] P:toolforge::bastion: install locales-all

https://gerrit.wikimedia.org/r/1020906

taavi claimed this task.

Now it’s working again, thanks a lot!