Add Python 3.9 to Wikimedia CI
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Legoktm
	Aug 19 2021, 1:09 AM

Description

Debian Bullseye comes with Python 3.9, so we're going to need support for it in CI (example: https://gerrit.wikimedia.org/r/c/labs/codesearch/+/713729). Currently we only support up to Python 3.8 via https://people.debian.org/~paravoid/python-all/

Presumably we can have a bullseye container that just runs tox with 3.9 for now until the python-all backports are available.

Python versions provided by Debian:

Distro	Python version
Stretch	3.4
Buster	3.7
Bullseye	3.9

Our CI images:

Image	Distro	Pythons
releng/tox	Stretch	3.4 3.5 3.6 3.7
releng/tox-buster	Buster	3.5 3.6 3.7 3.8

If we move CI to Buster we loose 3.4 which might not be an issue.

We can either:

backport 3.9 to Buster
build 3.5, 3.6, 3.7, 3.8 for Bullseye

Or drop support for python 3.4, 3.5 and 3.6 entirely and get the new minimum supported version to be 3.7 shipped by Buster but there bunch of repositories still relying on python 3.5 / Stretch.

Details

Subject	Repo	Branch	Lines +/-
jjb: update Python jobs for Python 3.9 distutils	integration/config	master	+18 -18
tox-buster: add python39-distutils to image	integration/config	master	+21 -0
jjb: Update Python jobs to ones with py39	integration/config	master	+18 -18
tox-buster: Add python3.9 to tox-buster base image	integration/config	master	+22 -0
tox.ini: explain why there are old Python versions	operations/cookbooks	master	+1 -0
aptrepo: create a component to backport python3.9 to unblock CI	operations/puppet	production	+1 -0
Add tox-bullseye container	integration/config	master	+276 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		None	T297789 Support bullseye/python 3.9
Resolved		jbond	T289222 Add Python 3.9 to Wikimedia CI
Resolved		hashar	T321724 Fails to build releng/tox-poetry-buster:0.0.1-s1
Resolved	BUG REPORT	dcaro	T321915 poetry is not working in tox-poetry-buster:0.0.2

Event Timeline

Legoktm created this task.Aug 19 2021, 1:09 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2021, 1:09 AM

Change 713972 had a related patch set uploaded (by Legoktm; author: Legoktm):

[integration/config@master] Add tox-bullseye container

https://gerrit.wikimedia.org/r/713972

gerritbot added a project: Patch-For-Review.Aug 20 2021, 4:09 AM

The CI images have http://apt.wikimedia.org/wikimedia buster-wikimedia thirdparty/pyall and Python 3.8 got added to it via T241195. Adding python 3.9 to it seems the easiest approach and would benefit all the repositories having the tox-docker job.

Other alternatives:

migrate the CI tox-docker job to Bullseye, in this case we would need a pyall component with Python 3.5 to 3.9.
for codesearch repository (and any repo that requires python3.9), migrate to Release Pipeline . That also lets one publish an image to our Docker registry and might open the path of migrating code search to k8s.

Can we keep the discussion in one place please? Either Gerrit or Phab, not both.

On Gerrit I wrote:

I not have the capacity to backport Python 3.9 to bullseye nor forward port older Python versions to bullseye, so I proposed adding this for just 3.9, which is missing in CI despite us using it elsewhere.
I hope this is an acceptable solution for needing Python 3.9 CI despite not being perfect.

In T289222#7342298, @hashar wrote:

The CI images have http://apt.wikimedia.org/wikimedia buster-wikimedia thirdparty/pyall and Python 3.8 got added to it via T241195. Adding python 3.9 to it seems the easiest approach and would benefit all the repositories having the tox-docker job.

I mean, if you think it's the easiest option, rather than the patch I wrote which already exists, please, be my guest to backport Python 3.9 to buster. That would be great.

migrate the CI tox-docker job to Bullseye, in this case we would need a pyall component with Python 3.5 to 3.9.

This seems like letting perfect be the enemy of the good. Migrating all of tox-docker over to bullseye is explicitly not what is being asked for.

for codesearch repository (and any repo that requires python3.9), migrate to Release Pipeline . That also lets one publish an image to our Docker registry and might open the path of migrating code search to k8s.

This seems unreasonable for every single repository we use that has Python. I'm marking this as high priority given that we have production code running on bullseye/Python 3.9, without any CI verifying it.

Legoktm mentioned this in T291978: Limit GitLab shared runners to images from Wikimedia Docker registry.Sep 29 2021, 10:18 PM

Legoktm mentioned this in T295001: Python CI fails when none of the interpreters in tox are available.Nov 4 2021, 4:15 PM

hashar updated the task description. (Show Details)Dec 9 2021, 1:12 PM

Change 713972 abandoned by Hashar:

[integration/config@master] Add tox-bullseye container

Reason:

We need either:

- Buster and backport 3.9
- Bullseye and build 3.5, 3.6, 3.7, 3.8 for Bullseye

Last week I have updated the task description T289222

https://gerrit.wikimedia.org/r/713972

Maintenance_bot removed a project: Patch-For-Review.Dec 13 2021, 1:10 PM

• Kormat mentioned this in T297789: Support bullseye/python 3.9.Dec 15 2021, 2:08 PM

• Kormat subscribed.

Xqt awarded a token.Jan 3 2022, 4:42 PM

Xqt added a parent task: T297789: Support bullseye/python 3.9.

Xqt subscribed.

See also T270389. 30% of Pywikibot pypi downloads are made for Python 3.9 (T266984) and it would be appropriate to have tests wit this Python version then.

Or drop support for python 3.4, 3.5 and 3.6 entirely and get the new minimum supported version to be 3.7

Why not? All these versions have reached their EOL. There is no syntax change except of the await/async keywords introduced in Python 3.7. Python 3.5 code will run with 3.7+ (except possibly if these keywords are used) whereas Python 3.7 code may not run with 3.5 (e.g. due to ordered dicts, f-strings, circular imports, new methods etc.)

@hashar is there any update on this? At the moment more than 30% of the production fleet runs bullseye.

Change 849034 had a related patch set uploaded (by Volans; author: Volans):

[operations/cookbooks@master] tox.ini: explain why there are old Python versions

https://gerrit.wikimedia.org/r/849034

gerritbot added a project: Patch-For-Review.Oct 25 2022, 9:36 AM

jbond subscribed.Oct 25 2022, 10:24 AM

Change 849049 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] aptrepo: create a component to backport python3.9 to unblock CI

https://gerrit.wikimedia.org/r/849049

Change 849049 merged by Jbond:

[operations/puppet@production] aptrepo: create a component to backport python3.9 to unblock CI

https://gerrit.wikimedia.org/r/849049

Change 849034 merged by jenkins-bot:

[operations/cookbooks@master] tox.ini: explain why there are old Python versions

https://gerrit.wikimedia.org/r/849034

Maintenance_bot removed a project: Patch-For-Review.Oct 26 2022, 3:31 PM

Change 849635 had a related patch set uploaded (by Jbond; author: John Bond):

[integration/config@master] tox-buster: Add python3.9 to tox-buster base image

https://gerrit.wikimedia.org/r/849635

gerritbot added a project: Patch-For-Review.Oct 26 2022, 6:07 PM

Or drop support for python 3.4, 3.5 and 3.6 entirely and get the new minimum supported version to be 3.7 shipped by Buster but there bunch of repositories still relying on python 3.5 / Stretch.

Theses are all EOL so I'm +1

Adding python 3.9 to it seems the easiest approach and would benefit all the repositories having the tox-docker job.

see: https://gerrit.wikimedia.org/r/849635

Change 849635 merged by jenkins-bot:

[integration/config@master] tox-buster: Add python3.9 to tox-buster base image

https://gerrit.wikimedia.org/r/849635

Maintenance_bot removed a project: Patch-For-Review.Oct 26 2022, 6:30 PM

hashar mentioned this in T321724: Fails to build releng/tox-poetry-buster:0.0.1-s1.Oct 26 2022, 6:36 PM

hashar closed subtask T321724: Fails to build releng/tox-poetry-buster:0.0.1-s1 as Resolved.Oct 26 2022, 6:51 PM

Thanks for the python3.9 package! I have build the CI images. Will switch the Jenkins jobs to it tomorrow

Change 849647 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] jjb: Update Python jobs to ones with py39

https://gerrit.wikimedia.org/r/849647

gerritbot added a project: Patch-For-Review.Oct 26 2022, 7:31 PM

Change 849647 merged by jenkins-bot:

[integration/config@master] jjb: Update Python jobs to ones with py39

https://gerrit.wikimedia.org/r/849647

Thanks @jbond !!!!!!!

Maintenance_bot removed a project: Patch-For-Review.Oct 27 2022, 7:30 AM

When adding py39 to a tox.ini CI fails due to lack of distutils.cmd (example: https://gerrit.wikimedia.org/r/850002).

distutils is deprecated and will be removed from Python. Debian already drops it and the stock packages from eg Debian Bullseye does not provide it in libpython3.9-stdlib. Instead one has to explicitly install python3.9-distutils.

The packages provided by buster-wikimedia thirdparty/pyall are build in a way that libpythonXX-stdlib does provide a full distutils:

$ dpkg -S /usr/lib/python3.8/distutils/cmd.py
libpython3.8-stdlib:amd64: /usr/lib/python3.8/distutils/cmd.py

But the built python3.9 does not provide it. One can potentially use python3.9-full which depends on eg python3.9-distutils but that packages has not been build and is not available in component/python3.9.

We would need to either:

build python3.9 the same way the pyall one got build which I guess got hacked in to include distutils?
also build python3.9-distutils and include it in thirdparty/python39