Page MenuHomePhabricator

running job with SGE doesn't get the right environment when using -once -continuous
Closed, DeclinedPublic

Description

When I run a python job using pywikibot it works with a simple jsub but fails to import pywikibot module when doing a jsub -once -continuous

It's a problem of a missing environment variable, PYTHONPATH

tools.phetools@tools-login:~$ cat .profile
. ~/.bashrc
export PYTHONPATH=/shared/pywikipedia/core:/shared/pywikipedia/core/externals/httplib2:/shared/pywikipedia/core/scripts

~/.bashrc contains also the same export

trying a simple

$ jsub -l h_vmem=512M -N test_env /bin/bash -c "env"

show PYTHONPATH in ~/test_env.out

but

$ jsub -once -continuous -l h_vmem=512M -N test_env /bin/bash -c "env"

doesn't show anything coming from ~/.profile or ~/.bashrc, same trouble with sh rather than /bin/bash.

results are in /var/data/project/test_env.out.good / /var/data/project/test_env.out.bad the .err is always empty


Version: unspecified
Severity: major

Details

Reference
bz65491

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:23 AM
bzimport added a project: Toolforge.
bzimport set Reference to bz65491.

typo, I meant /data/project/phetools/test_env.out.good and /data/project/phetools/test_env.out.bad

Looking the man page http://cf.ccmr.cornell.edu/cgi-bin/w3mman2html.cgi?qsub%281B%29 environment var are not exported to the job, it's the normal behavior, that docs says to use qsub -V to pass env variable, is it possible jsub doesn't pass -V to qsub when using -once -continuous ?

Actually, jsub passes -V in neither case; but the gridengine is really random about when it decides to start subshells or not. I strongly recommend that any dependency on the environment be sourced from the job script itself for that reason (or, possibly, by sourcing .bashrc explicitly as many have done).