For the adhs-wde tool I have two scripts: the webserver app.py running on kubernetes python3.7 and the synchronous worker planned to be run on the grid. Both have similar module requirements, in particular PyShEx>=0.7.14. I installed an environment via webservice --backend=kubernetes python3.7 shell and the app.py uses it successfully. However, I can not use it from toolserver bastion, viz:
rwst@tools-sgebastion-07:~$ become adhs-wde tools.adhs-wde@tools-sgebastion-07:~$ source www/python/venv/bin/activate (venv) tools.adhs-wde@tools-sgebastion-07:~$ which pip /data/project/adhs-wde/www/python/venv/bin/pip (venv) tools.adhs-wde@tools-sgebastion-07:~$ pip show pyshex Traceback (most recent call last): File "/data/project/adhs-wde/www/python/venv/bin/pip", line 5, in <module> from pip._internal.cli.main import main ImportError: No module named 'pip'
So I thought I'll create a specific environment for grid jobs.
become adhs-wde python3 -mvenv grid_env
The problem that I encountered was that, not only was a different python version installed but also that newer module versions were not available, e.g. PyShEx 0.7.10 vs 0.7.20 needed. Before I start to think about how to hack a better environment I wanted to ask if there is a more natural solution to the unavailability of new modules in the non-webserver environment. Also, should it not be possible to use the webserver environment for the grid, somehow?