e.g. current version https://github.com/SuLab/WikidataIntegrator/ only support Python 3.6+ (due to sparql-slurper dependency), but Toolforge only has Python 3.5.3. So it's not possible to run WikidataIntegrator without manually compile a newer version of Python.
The scope of this ticket is to add support for Python 3.7 in Toolforge Kubernetes
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Phamhi | T230961 Add Python 3.7 to Toolforge Kubernetes | |||
| Resolved | • Phamhi | T235629 Port toollabs-webservice to buster-tools |
stretch doesn't have python3.6 so as per suggestion from bstorm, I'm going to skip to python3.7 since it's available on buster
python37/web (specifically toollabs-webservice) needs python-pykube which doesn't look like it's available in buster
The following packages have unmet dependencies:
toollabs-webservice : Depends: python-pykube but it is not installable
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --yes toollabs-webservice uwsgi uwsgi-plugin-python3 && apt-get clean && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100
Traceback (most recent call last):
File "./build.py", line 228, in <module>
main()
File "./build.py", line 219, in main
args.tag,
File "./build.py", line 109, in build_image
subprocess.check_call(args)
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 347, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/docker', 'build', '-t', 'docker-registry.tools.wmflabs.org/toolforge-python37-web:testing', '--no-cache', '/Users/ll68349/wikimedia/toollabs-images/python37/web']' returned non-zero exit status 100.(buster docker instance) # apt policy python-pykube python-pykube: Installed: (none) Candidate: (none) Version table:
I found a couple of previous tickets related to this:
I think similar to https://phabricator.wikimedia.org/T200660, we need to forward-port python-pykube to buster. Will talk to @aborrero for help.
Change 543124 had a related patch set uploaded (by Phamhi; owner: Hieu Pham):
[operations/docker-images/toollabs-images@master] Update all images based on buster (T230961)
I might as well extend the scope of this ticket to update all docker images based on buster.
This means the following new language versions will be available:
Mentioned in SAL (#wikimedia-operations) [2019-10-15T12:13:08Z] <arturo> add copy of python-pykube and python3-pykube from stretch-wikimedia to buster-wikimedia (T230961)
It looks like toollabs-webservice is not in buster-tools... will ask for help from arturo again to port this package
Change 543124 had a related patch set uploaded (by Phamhi; owner: Hieu Pham):
[operations/docker-images/toollabs-images@master] Update all images based on buster.
Change 543124 merged by Phamhi:
[operations/docker-images/toollabs-images@master] Docker-images: create new docker images based on buster.
Mentioned in SAL (#wikimedia-cloud) [2019-11-04T11:10:24Z] <phamhi> Built and pushed python37 docker image based on buster (T230961)
Mentioned in SAL (#wikimedia-cloud) [2019-11-04T14:45:19Z] <phamhi> Built and pushed php73 docker image based on buster (T230961)
Mentioned in SAL (#wikimedia-cloud) [2019-11-04T14:45:30Z] <phamhi> Built and pushed jdk11 docker image based on buster (T230961)
Mentioned in SAL (#wikimedia-cloud) [2019-11-04T14:45:39Z] <phamhi> Built and pushed golang111 docker image based on buster (T230961)
Mentioned in SAL (#wikimedia-cloud) [2019-11-04T14:45:49Z] <phamhi> Built and pushed ruby25 docker image based on buster (T230961)
Change 549863 had a related patch set uploaded (by Phamhi; owner: Hieu Pham):
[operations/software/tools-webservice@master] tools-webservice: Add new buster image options to webservice
Updating tools-webservice to include new buster images options to webservice: golang111, jdk11, php73, python37 and ruby25
Change 549863 merged by jenkins-bot:
[operations/software/tools-webservice@master] tools-webservice: Add new buster image options to webservice
Mentioned in SAL (#wikimedia-cloud) [2019-11-08T18:37:37Z] <bstorm_> pushed new webservice package supporting buster containers to repo T230961
Mentioned in SAL (#wikimedia-cloud) [2019-11-08T18:40:16Z] <bstorm_> pushed new webservice package to the bastions T230961
Where is it installed? I'm still seeing only 3.5 on tools-sgebastion-08
$ ls -l /usr/bin/python3.* -rwxr-xr-x 2 root root 4751184 Sep 27 2018 /usr/bin/python3.5 lrwxrwxrwx 1 root root 33 Sep 27 2018 /usr/bin/python3.5-config -> x86_64-linux-gnu-python3.5-config -rwxr-xr-x 1 root root 401 Sep 24 2016 /usr/bin/python3.5-coverage -rwxr-xr-x 2 root root 4751184 Sep 27 2018 /usr/bin/python3.5m lrwxrwxrwx 1 root root 34 Sep 27 2018 /usr/bin/python3.5m-config -> x86_64-linux-gnu-python3.5m-config
Hi @RoySmith. Python 3.7 can be found if you start your webservice using kubernetes backend.
For example:
tools.phamhi-tool@tools-sgebastion-07:~$ webservice --backend=kubernetes python3.7 shell Defaulting container name to interactive. Use 'kubectl describe pod/interactive -n phamhi-tool' to see all of the containers in this pod. If you don't see a command prompt, try pressing enter. tools.phamhi-tool@interactive:~$ python3 --version Python 3.7.3
Can it be made available from the command line on the bastion hosts? And in the grid environment?
Hi @RoySmith , our bastion servers are on debian stretch which only has up to Python 3.5; we would have to upgrade to Buster which is not on our roadmap so I wouldn't be holding my breath if I were you.
When you invoke webservice --backend=kubernetes python3.7 shell, the container maps to your home directory so there's literally no difference between the Python 3.7 shell environment and your normal shell environment. Is there something in particular that is not working?
Hmmm. I've only been running in the grid environment, but it sounds like I should be looking at moving over the k8s in general. I'm reading up on that.
In any case, the command you give doesn't work for me:
$ webservice --backend=kubernetes python3.7 shell Traceback (most recent call last): File "/usr/local/bin/webservice", line 134, in <module> tool = Tool.from_currentuser() File "/usr/lib/python2.7/dist-packages/toollabs/common/tool.py", line 97, in from_currentuser return Tool.from_pwd(pwd_entry) File "/usr/lib/python2.7/dist-packages/toollabs/common/tool.py", line 106, in from_pwd 'Tool username should begin with ' + Tool.PREFIX) toollabs.common.tool.InvalidToolException: Tool username should begin with tools.
I'm not 100% sure what question you're asking. I login as user "roysmith", and then run "become roysmith-test". I guess the later is what you're looking for?
That sounds like you tried to run the command as roysmith before running become roysmith-test. It can only be run as the tool account after you use the become command.
Updated to what was actually done since python 3.7 isn't available (and isn't going to be for some time) on the bastions or grid