$ sudo become integraality
$ ls -1 www/python/venv/bin/ | grep '3.'
easy_install-3.7
pip3.7

This python3.7 venv can only be used from a runtime that has python3.7 installed. In Toolforge, that means you must be operating from inside a python3.7 container on the Kubernetes cluster.

Your options are:

1. Roll back to python3.5 which is available on both the Kubernetes cluster and the Grid Engine cluster
2. Replace your crontab jobs running on the Grid Engine with Kubernetes CronJob objects

In T257942#6305237, @bd808 wrote:

Your options are:

1. Roll back to python3.5 which is available on both the Kubernetes cluster and the Grid Engine cluster
2. Replace your crontab jobs running on the Grid Engine with Kubernetes CronJob objects

3. Build a second venv using python3.5 just for use on the Grid Engine cluster

So:

• crontab invokes jsub with the run.sh script (jsub -mem 1000m -once -j y -o /data/project/integraality/logs/update.log -N update /data/project/integraality/integraality/bin/run.sh)
• run.sh:
  • sources the virtual environment (via /data/project/integraality/www/python//venv/bin/activate)
  • runs the python script (via python integraality/pages_processor.py)

I tried changing to run VIRTUAL_ENV_PATH/bin/python3 integraality/pages_processor.py (as stated in Help:Toolforge/Python#Use_venv_with_scheduled_tasks ) but the same happens.

I tried slapping a pip freeze in run.sh (after sourcing the venv) to see what happens:

+ pip freeze
Traceback (most recent call last):
  File "/data/project/integraality/www/python/venv/bin/pip", line 6, in <module>
    from pip._internal.cli.main import main
ImportError: No module named 'pip'

In T257942#6305256, @JeanFred wrote:

I tried changing to run VIRTUAL_ENV_PATH/bin/python3 integraality/pages_processor.py (as stated in Help:Toolforge/Python#Use_venv_with_scheduled_tasks ) but the same happens.

That documentation is badly written. It was true in the long ago when there was not a Kubernetes cluster and all venvs were built with the bastion/grid engine Python interpreter. It is not true when you are mixing Kubernetes and Grid Engine. You need to do one of the 3 things from T257942#6305239.

Thanks @bd808 for the investigation and hints!

In T257942#6305237, @bd808 wrote:

This python3.7 venv can only be used from a runtime that has python3.7 installed. In Toolforge, that means you must be operating from inside a python3.7 container on the Kubernetes cluster.

Hmm, clearly I misunderstood, but when I read the announcements about py37 and the 2020 k8s cluster, I was not clear to me that this would break my scheduled jobs as well − of course, that was stupid of me because jsub jobs are by definition Grid-Engine based, not k8s-based, but still…

3. Build a second venv using python3.5 just for use on the Grid Engine cluster

I already went for option 3 as a stopgap measure ; and can confirm that the update ran overnight. I will look into option 2 k8s CronJob objects at a later point.

Mentioned in SAL (#wikimedia-cloud) [2021-04-03T16:44:58Z] <wm-bot> <jeanfred> Deploy latest from Git master: eac4d2c (T257942)

Mentioned in SAL (#wikimedia-cloud) [2021-04-03T21:33:33Z] <wm-bot> <jeanfred> Deploy latest from Git master: eac4d2c (T257942)

Mentioned in SAL (#wikimedia-cloud) [2022-05-14T14:38:03Z] <wm-bot> <jeanfred> Deploy 2a521d1, 29d6b62, 8bd793d (T257942)