Page MenuHomePhabricator

Confusion about the os & package versions used on Toolforge bastions & grid vs Kubernetes containers
Closed, ResolvedPublicBUG REPORT

Description

For the adhs-wde tool I have two scripts: the webserver app.py running on kubernetes python3.7 and the synchronous worker planned to be run on the grid. Both have similar module requirements, in particular PyShEx>=0.7.14. I installed an environment via webservice --backend=kubernetes python3.7 shell and the app.py uses it successfully. However, I can not use it from toolserver bastion, viz:

rwst@tools-sgebastion-07:~$ become adhs-wde
tools.adhs-wde@tools-sgebastion-07:~$ source www/python/venv/bin/activate
(venv) tools.adhs-wde@tools-sgebastion-07:~$ which pip
/data/project/adhs-wde/www/python/venv/bin/pip
(venv) tools.adhs-wde@tools-sgebastion-07:~$ pip show pyshex
Traceback (most recent call last):
  File "/data/project/adhs-wde/www/python/venv/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ImportError: No module named 'pip'

So I thought I'll create a specific environment for grid jobs.

become adhs-wde
python3 -mvenv grid_env

The problem that I encountered was that, not only was a different python version installed but also that newer module versions were not available, e.g. PyShEx 0.7.10 vs 0.7.20 needed. Before I start to think about how to hack a better environment I wanted to ask if there is a more natural solution to the unavailability of new modules in the non-webserver environment. Also, should it not be possible to use the webserver environment for the grid, somehow?

Event Timeline

SCIdude renamed this task from python env missing newer packages to Toolforge python env missing newer packages.Feb 1 2021, 5:19 PM

Python 3.5 is used in Debian Stretch (which the grid uses), and that has been deprecated by the python project https://www.python.org/dev/peps/pep-0478/. It is quite common for more recent modules to drop support during upgrades for it now (including pip itself). We are working on introducing Debian Buster grid nodes in the near future. When I'm sure of phab tasks around that, I can link you to them.

@aborrero If/when we have a task for buster bastions and compute nodes, I suspect this is effectively a "duplicate" as are any tasks to upgrade to python 3.7 in Toolforge that may be floating about.

@SCIdude Beyond that information, if you need stuff right away, you may have to experiment with pyenv or something for gridengine in order to install a later python on your tool's home directory, unfortunately.

That explains it, thanks. Would it be odd to run a (low key) worker in a kubernetes box?

It would be unusual right now, but we are, in fact, moving in that direction as well (in the design phase right now T251917). The goal being to make K8s jobs easier to manage. We have some people running jobs in Kubernetes now, but you have to do self-service management with kubectl. I understand that CronJobs objects are bit easier to manage because you can set a maximum number of finished jobs, and the controller cleans up for you. That issue with Job objects came up in T251027. There is a label that you should apply to the pod spec in any such object that will make sure your pod will mount the NFS folders you need toolforge": "tool
That should magically make the NFS appear in your pod.

If you want to give the CronJobs object a shot now, the upstream docs are here: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
Quotas limit the number of pods you can have, so you'll want to balance the needs of interactive pods (webservice shell), webservice pods and cron pods. If you need us to increase a quota, we have a process to do that and docs here https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Quotas_and_Resources. Additionally, if you do go that route, I'd encourage you to document experiences in phabricator or somewhere here https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes so we can benefit from it where people are willing to brave that new world.

We also have a small number of tools that run worker processes in Kubernetes in a manually managed service as well. However, if you just need regularly scheduled jobs, the CronJob object might be a better fit. When the whole service is built, we'll have some better tooling to help. That will be a while, though.

bd808 renamed this task from Toolforge python env missing newer packages to Confusion about the os & package versions used on Toolforge bastions & grid vs Kubernetes containers.Feb 2 2021, 9:16 PM
Majavah added a subscriber: Majavah.

Closing this as the question itself was answered and there are other open tasks tracking about various updates.