For the `adhs-wde` tool I have two scripts: the webserver `app.py` running on `kubernetes python3.7` and the synchronous worker planned to be run on the grid. Both have similar module requirements, in particular `PyShEx>=0.7.14`. I installed an environment via `webservice --backend=kubernetes python3.7 shell` and the `app.py` uses it. However, I can not use it from toolserver bastion, viz:
rwst@tools-sgebastion-07:~$ become adhs-wde
tools.adhs-wde@tools-sgebastion-07:~$ source www/python/venv/bin/activate
(venv) tools.adhs-wde@tools-sgebastion-07:~$ which pip
(venv) tools.adhs-wde@tools-sgebastion-07:~$ pip show pyshex
Traceback (most recent call last):
File "/data/project/adhs-wde/www/python/venv/bin/pip", line 5, in <module>
from pip._internal.cli.main import main
ImportError: No module named 'pip'
So I thought I'll create a specific environment for grid jobs.
python3 -mvenv grid_env
The problem that I encountered was that, not only was a different python version installed but also that newer module versions were not available, e.g. PyShEx 0.7.10 vs 0.7.20 needed. Before I start to think about how to hack a better environment I wanted to ask if there is a more natural solution to the unavailability of new modules in the non-webserver environment. Also, should it not be possible to use the webserver environment for the grid, somehow?