Page MenuHomePhabricator

Inconsistent locale settings of different grid compute nodes.
Closed, ResolvedPublic

Description

For T121305, I've set up a cron job to check for jsubmissions every 5 minutes. The output is from a simple date | tee jsubtest.txt >&2, yet it generate output two different locales on different runs:

Eg.

Tue Dec 15 10:00:16 UTC 2015
Di 15. Dez 10:05:05 UTC 2015

The majority belongs to the former (English) locale. The latter I cannot identify.

Event Timeline

zhuyifei1999 raised the priority of this task from to Low.
zhuyifei1999 updated the task description. (Show Details)
zhuyifei1999 added a project: Toolforge.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptDec 15 2015, 10:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
*/5 * * * * jsub -quiet -once -N jsubtest sh test.sh
valhallasw@tools-bastion-01:/data/project/yifeibot$ cat test.sh
#! /bin/sh
date | tee jsubtest.txt >&2

I've been so free to add

hostname | tee jsubtest.txt >&2

so the hostname is also logged.

Stigmj added a subscriber: Stigmj.Dec 15 2015, 12:05 PM

I have seen these as well.. Seems to be a german locale:

rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: das Entfernen von »/data/project/pagecount/total-import-is-running“ ist nicht möglich: Datei oder Verzeichnis nicht gefunden
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: das Entfernen von »/data/project/pagecount/total-import-is-running“ ist nicht möglich: Datei oder Verzeichnis nicht gefunden
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory
rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory

Very strange. It seems connected to tools-exec-1201:

(from tools.yifeibot)

tools-exec-1211: Tue Dec 15 11:30:12 UTC 2015
tools-exec-1219: Tue Dec 15 11:35:05 UTC 2015
tools-exec-1201: Di 15. Dez 11:50:07 UTC 2015

There are some other oddities as well. For example, when I test from the console:

tools-exec-1204: Tue Dec 15 12:08:10 UTC 2015
LANG=

while

tools-exec-1220: Tue Dec 15 12:08:14 UTC 2015
LANG=en_US.UTF-8

(but that might be because I'm testing this directly from the console/tools-bastion-01).

valhallasw@tools-exec-1201:/var/spool/gridengine/execd/tools-exec-1201$ sudo less /proc/10174/environ
"/proc/10174/environ" may be a binary file.  See it anyway?
(...) LANG=de_DE.UTF-8

So.. it's de_DE.UTF-8 and it happens for multiple users, but only on tools-exec-1201. All other hosts just have LANG=C and LANG=en_US.UTF-8. Huh?

de_DE.UTF-8 /is/ installed on other hosts:

valhallasw@tools-exec-1220:~$ LANG=de_DE.UTF-8 ls fdfdsaf
ls: Zugriff auf fdfdsaf nicht möglich: Datei oder Verzeichnis nicht gefunden

Not really relevant, but I changed the script to

echo `hostname`: `date` | tee jsubtest.txt >&2

for the sake of outputting in a single line.

valhallasw moved this task from Triage to Backlog on the Toolforge board.Dec 22 2015, 12:08 PM
zhuyifei1999 closed this task as Resolved.Jan 10 2016, 10:59 AM
zhuyifei1999 claimed this task.

Not sure why & how, but after the outage on T122638: GridEngine down due to bdb issues (and a reboot of the instance) the issue seems gone:

tools.yifeibot@tools-bastion-02:~$ grep 1201 jsubtest.err | tail
tools-exec-1201: Di 29. Dez 16:10:07 UTC 2015
tools-exec-1201: Di 29. Dez 18:55:06 UTC 2015
tools-exec-1201: Di 29. Dez 20:10:08 UTC 2015
tools-exec-1201: Mon Jan 4 21:00:19 UTC 2016
tools-exec-1201: Wed Jan 6 04:15:07 UTC 2016
tools-exec-1201: Wed Jan 6 10:00:19 UTC 2016
tools-exec-1201: Wed Jan 6 12:00:23 UTC 2016
tools-exec-1201: Wed Jan 6 16:00:15 UTC 2016
tools-exec-1201: Thu Jan 7 04:05:06 UTC 2016
tools-exec-1201: Fri Jan 8 00:05:06 UTC 2016
zhuyifei1999 removed zhuyifei1999 as the assignee of this task.Jan 10 2016, 11:00 AM
zhuyifei1999 set Security to None.