Python 3.5 is used in Debian Stretch (which the grid uses), and that has been deprecated by the python project https://www.python.org/dev/peps/pep-0478/. It is quite common for more recent modules to drop support during upgrades for it now (including pip itself). We are working on introducing Debian Buster grid nodes in the near future. When I'm sure of phab tasks around that, I can link you to them.

@aborrero If/when we have a task for buster bastions and compute nodes, I suspect this is effectively a "duplicate" as are any tasks to upgrade to python 3.7 in Toolforge that may be floating about.

@SCIdude Beyond that information, if you need stuff right away, you may have to experiment with pyenv or something for gridengine in order to install a later python on your tool's home directory, unfortunately.

That explains it, thanks. Would it be odd to run a (low key) worker in a kubernetes box?

It would be unusual right now, but we are, in fact, moving in that direction as well (in the design phase right now T251917). The goal being to make K8s jobs easier to manage. We have some people running jobs in Kubernetes now, but you have to do self-service management with kubectl. I understand that CronJobs objects are bit easier to manage because you can set a maximum number of finished jobs, and the controller cleans up for you. That issue with Job objects came up in T251027. There is a label that you should apply to the pod spec in any such object that will make sure your pod will mount the NFS folders you need toolforge": "tool
That should magically make the NFS appear in your pod.

If you want to give the CronJobs object a shot now, the upstream docs are here: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
Quotas limit the number of pods you can have, so you'll want to balance the needs of interactive pods (webservice shell), webservice pods and cron pods. If you need us to increase a quota, we have a process to do that and docs here https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Quotas_and_Resources. Additionally, if you do go that route, I'd encourage you to document experiences in phabricator or somewhere here https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes so we can benefit from it where people are willing to brave that new world.

We also have a small number of tools that run worker processes in Kubernetes in a manually managed service as well. However, if you just need regularly scheduled jobs, the CronJob object might be a better fit. When the whole service is built, we'll have some better tooling to help. That will be a while, though.

Very helpful, thanks again.