Mentioned in SAL (#wikimedia-cloud) [2019-07-22T20:55:04Z] <lucaswerkmeister> Deployed cb848cc4a7 (fix PATH specification, working around T228696)

Nothing has changed in our jsub wrapper that would stop -v ... from being passed to the qsub binary. man qsub still shows -v variable[=value],... as a method for altering a job's environment. It looks like adding arbitrary environment variables using -v ... works as expected in both @LucasWerkmeister's example and my tests as well.

The PATH that @LucasWerkmeister is showing in his output however matches the one that /etc/profile sets unconditionally on our Stretch hosts. I wonder if what happens in this case is that qsub does pass the environment variable to the spawned job, but that happens before the process is started and the submitting user's profile overrides it? I guess I would need to dig into what exactly the sge_shepard process does to launch a job to confirm or deny this theory.

If it’s due to /etc/profile, then bypassing the shell might do the trick? I copied /usr/bin/jsub into my home directory and patched it to pass through qsub’s -shell option (P8781), and the results were… well…

lucaswerkmeister@tools-sgebastion-07:~$ ./jsub -sync y -shell n -v PATH env
Your job 6188896 ("env") has been submitted
Job 6188896 exited with exit code 0.
lucaswerkmeister@tools-sgebastion-07:~$ grep ^PATH= env.out                                                                                                                                                                                                                                                                                                
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
PATH=/tmp/6188896.1.task:/home/lucaswerkmeister/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

I don’t understand why there are two PATHs (the first one is the one from /etc/profile, which is not the same as the one documented in the qsub manpage as the default: that would be /usr/local/bin:/usr/ucb:/bin:/usr/bin), nor why an extra directory appears at the beginning of the second one.

In T228696#5355919, @LucasWerkmeister wrote:

I don’t understand why there are two PATHs (the first one is the one from /etc/profile, which is not the same as the one documented in the qsub manpage as the default: that would be /usr/local/bin:/usr/ucb:/bin:/usr/bin), nor why an extra directory appears at the beginning of the second one.

Did you delete the env.out from the prior test before this one? If you did not then that might explain the double PATH output as the log files are written in append mode rather than replace.

The prepended path component... maybe that's the job's working directory on the exec node? Does it happen to match the value of SGE_JOB_SPOOL_DIR from the env dump?

Yup, double PATH was due to not deleting the output file, thanks. (P8783 is a fixed version.) The extra directory is not the SGE_JOB_SPOOL_DIR (that’s somewhere in /var/spool/, though it ends with the same number), but it’s the same as TMP and TMPDIR.

I tried a few more combinations of options:

lucaswerkmeister@tools-sgebastion-07:~$ export PATH=/EASILY/RECOGNIZABLE/CUSTOM/PATH:$PATH
lucaswerkmeister@tools-sgebastion-07:~$ for var in '' '-v PATH'; do for shell in '-shell n' '-shell y'; do echo ./jsub -sync y -quiet $var $shell env; ./jsub -sync y -quiet $var $shell env; grep ^PATH= env.out; rm env.{out,err}; done; done
./jsub -sync y -quiet -shell n env
PATH=/tmp/6212946.1.task:/usr/local/bin:/bin:/usr/bin
./jsub -sync y -quiet -shell y env
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
./jsub -sync y -quiet -v PATH -shell n env
PATH=/tmp/6212963.1.task:/EASILY/RECOGNIZABLE/CUSTOM/PATH:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
./jsub -sync y -quiet -v PATH -shell y env
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

It looks like with -shell y, the PATH in /etc/profile indeed overrides the one passed by the grid, because then the TMPDIR is gone as well. And with -shell n and -v PATH, the PATH is inherited as it should be. (I left out omitting the -shell option because that just tests the default of my patched jsub, not of the underlying qsub.)