Change Details

For the `adhs-wde` tool I have two scripts: the webserver `app.py` running on `kubernetes python3.7` and the synchronous worker planned to be run on the grid. Both have similar module requirements, in particular `PyShEx>=0.7.14`. I installed an environment via `webservice --backend=kubernetes python3.7 shell` and the `app.py` uses it. However, I can not use it from toolserver bastion, viz: ``` rwst@tools-sgebastion-07:~$ become adhs-wde tools.adhs-wde@tools-sgebastion-07:~$ source www/python/venv/bin/activate (venv) tools.adhs-wde@tools-sgebastion-07:~$ which pip /data/project/adhs-wde/www/python/venv/bin/pip (venv) tools.adhs-wde@tools-sgebastion-07:~$ pip show pyshex Traceback (most recent call last): File "/data/project/adhs-wde/www/python/venv/bin/pip", line 5, in <module> from pip._internal.cli.main import main ImportError: No module named 'pip' ``` So I thought I'll create a specific environment for grid jobs. ``` become adhs-wde python3 -mvenv grid_env ``` The problem that I encountered was that, not only was a different python version installed but also that newer module versions were not available, e.g. PyShEx 0.7.10 vs 0.7.20 needed. Before I start to think about how to hack a better environment I wanted to ask if there is a more natural solution to the unavailability of new modules in the non-webserver environment. Also, should it not be possible to use the webserver environment for the grid, somehow?

For the `adhs-wde` tool I have two scripts: the webserver `app.py` running on `kubernetes python3.7` and the synchronous worker planned to be run on the grid. Both have similar module requirements, in particular `PyShEx>=0.7.14`. I installed an environment via `webservice --backend=kubernetes python3.7 shell` and the `app.py` uses it successfully. However, I can not use it from toolserver bastion, viz: ``` rwst@tools-sgebastion-07:~$ become adhs-wde tools.adhs-wde@tools-sgebastion-07:~$ source www/python/venv/bin/activate (venv) tools.adhs-wde@tools-sgebastion-07:~$ which pip /data/project/adhs-wde/www/python/venv/bin/pip (venv) tools.adhs-wde@tools-sgebastion-07:~$ pip show pyshex Traceback (most recent call last): File "/data/project/adhs-wde/www/python/venv/bin/pip", line 5, in <module> from pip._internal.cli.main import main ImportError: No module named 'pip' ``` So I thought I'll create a specific environment for grid jobs. ``` become adhs-wde python3 -mvenv grid_env ``` The problem that I encountered was that, not only was a different python version installed but also that newer module versions were not available, e.g. PyShEx 0.7.10 vs 0.7.20 needed. Before I start to think about how to hack a better environment I wanted to ask if there is a more natural solution to the unavailability of new modules in the non-webserver environment. Also, should it not be possible to use the webserver environment for the grid, somehow?

For the `adhs-wde` tool I have two scripts: the webserver `app.py` running on `kubernetes python3.7` and the synchronous worker planned to be run on the grid. Both have similar module requirements, in particular `PyShEx>=0.7.14`. I installed an environment via `webservice --backend=kubernetes python3.7 shell` and the `app.py` uses it successfully. However, I can not use it from toolserver bastion, viz: ``` rwst@tools-sgebastion-07:~$ become adhs-wde tools.adhs-wde@tools-sgebastion-07:~$ source www/python/venv/bin/activate (venv) tools.adhs-wde@tools-sgebastion-07:~$ which pip /data/project/adhs-wde/www/python/venv/bin/pip (venv) tools.adhs-wde@tools-sgebastion-07:~$ pip show pyshex Traceback (most recent call last): File "/data/project/adhs-wde/www/python/venv/bin/pip", line 5, in <module> from pip._internal.cli.main import main ImportError: No module named 'pip' ``` So I thought I'll create a specific environment for grid jobs. ``` become adhs-wde python3 -mvenv grid_env ``` The problem that I encountered was that, not only was a different python version installed but also that newer module versions were not available, e.g. PyShEx 0.7.10 vs 0.7.20 needed. Before I start to think about how to hack a better environment I wanted to ask if there is a more natural solution to the unavailability of new modules in the non-webserver environment. Also, should it not be possible to use the webserver environment for the grid, somehow?