Page MenuHomePhabricator

Add Python 3.7 to Toolforge Kubernetes
Closed, ResolvedPublic

Description

e.g. current version https://github.com/SuLab/WikidataIntegrator/ only support Python 3.6+ (due to sparql-slurper dependency), but Toolforge only has Python 3.5.3. So it's not possible to run WikidataIntegrator without manually compile a newer version of Python.

The scope of this ticket is to add support for Python 3.7 in Toolforge Kubernetes

Details

Related Gerrit Patches:
operations/software/tools-webservice : mastertools-webservice: Add new buster image options to webservice
operations/docker-images/toollabs-images : masterDocker-images: create new docker images based on buster.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 21 2019, 10:31 PM
Aklapper renamed this task from Install newer version of Python in Toolforge to Install a version of Python newer than 3.5.3 in Toolforge.Aug 21 2019, 11:03 PM
Phamhi claimed this task.Sep 30 2019, 8:09 PM

stretch doesn't have python3.6 so as per suggestion from bstorm, I'm going to skip to python3.7 since it's available on buster

Phamhi added a comment.EditedOct 11 2019, 9:30 PM

python37/web (specifically toollabs-webservice) needs python-pykube which doesn't look like it's available in buster

The following packages have unmet dependencies:
 toollabs-webservice : Depends: python-pykube but it is not installable
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c apt-get update     && DEBIAN_FRONTEND=noninteractive     apt-get install --yes     toollabs-webservice     uwsgi     uwsgi-plugin-python3     && apt-get clean     && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100
Traceback (most recent call last):
  File "./build.py", line 228, in <module>
    main()
  File "./build.py", line 219, in main
    args.tag,
  File "./build.py", line 109, in build_image
    subprocess.check_call(args)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/docker', 'build', '-t', 'docker-registry.tools.wmflabs.org/toolforge-python37-web:testing', '--no-cache', '/Users/ll68349/wikimedia/toollabs-images/python37/web']' returned non-zero exit status 100.
(buster docker instance) # apt policy python-pykube
python-pykube:
  Installed: (none)
  Candidate: (none)
  Version table:

I think similar to https://phabricator.wikimedia.org/T200660, we need to forward-port python-pykube to buster. Will talk to @aborrero for help.

Change 543124 had a related patch set uploaded (by Phamhi; owner: Hieu Pham):
[operations/docker-images/toollabs-images@master] Update all images based on buster (T230961)

https://gerrit.wikimedia.org/r/543124

I might as well extend the scope of this ticket to update all docker images based on buster.

This means the following new language versions will be available:

  • Golang 1.11
  • JDK 11
  • PHP 7.3
  • Python 3.7
  • Ruby2.5
  • (possibly more to come)

Mentioned in SAL (#wikimedia-operations) [2019-10-15T12:13:08Z] <arturo> add copy of python-pykube and python3-pykube from stretch-wikimedia to buster-wikimedia (T230961)

Phamhi added a comment.EditedOct 15 2019, 12:28 PM

Woot woot... confirmed.. thanks arturo for unblocking me

It looks like toollabs-webservice is not in buster-tools... will ask for help from arturo again to port this package

Change 543124 had a related patch set uploaded (by Phamhi; owner: Hieu Pham):
[operations/docker-images/toollabs-images@master] Update all images based on buster.

https://gerrit.wikimedia.org/r/543124

Change 543124 merged by Phamhi:
[operations/docker-images/toollabs-images@master] Docker-images: create new docker images based on buster.

https://gerrit.wikimedia.org/r/543124

ST47 added a subscriber: ST47.Nov 1 2019, 11:45 PM

Mentioned in SAL (#wikimedia-cloud) [2019-11-04T11:10:24Z] <phamhi> Built and pushed python37 docker image based on buster (T230961)

Mentioned in SAL (#wikimedia-cloud) [2019-11-04T14:45:19Z] <phamhi> Built and pushed php73 docker image based on buster (T230961)

Mentioned in SAL (#wikimedia-cloud) [2019-11-04T14:45:30Z] <phamhi> Built and pushed jdk11 docker image based on buster (T230961)

Mentioned in SAL (#wikimedia-cloud) [2019-11-04T14:45:39Z] <phamhi> Built and pushed golang111 docker image based on buster (T230961)

Mentioned in SAL (#wikimedia-cloud) [2019-11-04T14:45:49Z] <phamhi> Built and pushed ruby25 docker image based on buster (T230961)

Change 549863 had a related patch set uploaded (by Phamhi; owner: Hieu Pham):
[operations/software/tools-webservice@master] tools-webservice: Add new buster image options to webservice

https://gerrit.wikimedia.org/r/549863

Phamhi added a comment.Nov 8 2019, 3:31 PM

Updating tools-webservice to include new buster images options to webservice: golang111, jdk11, php73, python37 and ruby25

Phamhi added a comment.Nov 8 2019, 3:57 PM

I tested these images and confirmed working.

Change 549863 merged by jenkins-bot:
[operations/software/tools-webservice@master] tools-webservice: Add new buster image options to webservice

https://gerrit.wikimedia.org/r/549863

Mentioned in SAL (#wikimedia-cloud) [2019-11-08T18:37:37Z] <bstorm_> pushed new webservice package supporting buster containers to repo T230961

Mentioned in SAL (#wikimedia-cloud) [2019-11-08T18:40:16Z] <bstorm_> pushed new webservice package to the bastions T230961

aborrero triaged this task as Medium priority.Fri, Nov 22, 10:15 AM
Phamhi renamed this task from Install a version of Python newer than 3.5.3 in Toolforge to Add Python 3.7 in Toolforge.Mon, Dec 9, 1:23 PM
Phamhi closed this task as Resolved.
Phamhi updated the task description. (Show Details)

Python 3.7 support has been added.

Python 3.7 support has been added.

Where is it installed? I'm still seeing only 3.5 on tools-sgebastion-08

$ ls -l /usr/bin/python3.*
-rwxr-xr-x 2 root root 4751184 Sep 27  2018 /usr/bin/python3.5
lrwxrwxrwx 1 root root      33 Sep 27  2018 /usr/bin/python3.5-config -> x86_64-linux-gnu-python3.5-config
-rwxr-xr-x 1 root root     401 Sep 24  2016 /usr/bin/python3.5-coverage
-rwxr-xr-x 2 root root 4751184 Sep 27  2018 /usr/bin/python3.5m
lrwxrwxrwx 1 root root      34 Sep 27  2018 /usr/bin/python3.5m-config -> x86_64-linux-gnu-python3.5m-config
Phamhi added a comment.Mon, Dec 9, 3:26 PM

Hi @RoySmith. Python 3.7 can be found if you start your webservice using kubernetes backend.

For example:

tools.phamhi-tool@tools-sgebastion-07:~$ webservice --backend=kubernetes python3.7 shell
Defaulting container name to interactive.
Use 'kubectl describe pod/interactive -n phamhi-tool' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.

tools.phamhi-tool@interactive:~$ python3 --version
Python 3.7.3
RoySmith added a comment.EditedMon, Dec 9, 3:28 PM

Can it be made available from the command line on the bastion hosts? And in the grid environment?

Phamhi added a comment.Mon, Dec 9, 3:55 PM

Hi @RoySmith , our bastion servers are on debian stretch which only has up to Python 3.5; we would have to upgrade to Buster which is not on our roadmap so I wouldn't be holding my breath if I were you.

When you invoke webservice --backend=kubernetes python3.7 shell, the container maps to your home directory so there's literally no difference between the Python 3.7 shell environment and your normal shell environment. Is there something in particular that is not working?

Hmmm. I've only been running in the grid environment, but it sounds like I should be looking at moving over the k8s in general. I'm reading up on that.

In any case, the command you give doesn't work for me:

$  webservice --backend=kubernetes python3.7 shell
Traceback (most recent call last):
  File "/usr/local/bin/webservice", line 134, in <module>
    tool = Tool.from_currentuser()
  File "/usr/lib/python2.7/dist-packages/toollabs/common/tool.py", line 97, in from_currentuser
    return Tool.from_pwd(pwd_entry)
  File "/usr/lib/python2.7/dist-packages/toollabs/common/tool.py", line 106, in from_pwd
    'Tool username should begin with ' + Tool.PREFIX)
toollabs.common.tool.InvalidToolException: Tool username should begin with tools.
Phamhi added a comment.Mon, Dec 9, 6:19 PM

What's your tool's account?

I'm not 100% sure what question you're asking. I login as user "roysmith", and then run "become roysmith-test". I guess the later is what you're looking for?

Bstorm added a subscriber: Bstorm.Mon, Dec 9, 7:46 PM

That sounds like you tried to run the command as roysmith before running become roysmith-test. It can only be run as the tool account after you use the become command.

Ah, got it. Works fine now, and python 3.7.3 is indeed there. Thanks!

JJMC89 renamed this task from Add Python 3.7 in Toolforge to Add Python 3.7 to Toolforge Kubernetes.Tue, Dec 10, 1:53 AM
JJMC89 updated the task description. (Show Details)
JJMC89 added a subscriber: JJMC89.Tue, Dec 10, 1:56 AM

Updated to what was actually done since python 3.7 isn't available (and isn't going to be for some time) on the bastions or grid