Page MenuHomePhabricator

Add Python 3.9 to Wikimedia CI
Open, HighPublic

Description

Debian Bullseye comes with Python 3.9, so we're going to need support for it in CI (example: https://gerrit.wikimedia.org/r/c/labs/codesearch/+/713729). Currently we only support up to Python 3.8 via https://people.debian.org/~paravoid/python-all/

Presumably we can have a bullseye container that just runs tox with 3.9 for now until the python-all backports are available.

Python versions provided by Debian:

DistroPython version
Stretch3.4
Buster3.7
Bullseye3.9

Our CI images:

ImageDistroPythons
releng/toxStretch3.4 3.5 3.6 3.7
releng/tox-busterBuster3.5 3.6 3.7 3.8

If we move CI to Buster we loose 3.4 which might not be an issue.

We can either:

  • backport 3.9 to Buster
  • build 3.5, 3.6, 3.7, 3.8 for Bullseye

Or drop support for python 3.4, 3.5 and 3.6 entirely and get the new minimum supported version to be 3.7 shipped by Buster but there bunch of repositories still relying on python 3.5 / Stretch.

Event Timeline

Change 713972 had a related patch set uploaded (by Legoktm; author: Legoktm):

[integration/config@master] Add tox-bullseye container

https://gerrit.wikimedia.org/r/713972

The CI images have http://apt.wikimedia.org/wikimedia buster-wikimedia thirdparty/pyall and Python 3.8 got added to it via T241195. Adding python 3.9 to it seems the easiest approach and would benefit all the repositories having the tox-docker job.

Other alternatives:

  • migrate the CI tox-docker job to Bullseye, in this case we would need a pyall component with Python 3.5 to 3.9.
  • for codesearch repository (and any repo that requires python3.9), migrate to Release Pipeline . That also lets one publish an image to our Docker registry and might open the path of migrating code search to k8s.
Legoktm triaged this task as High priority.Sep 15 2021, 5:49 PM

Can we keep the discussion in one place please? Either Gerrit or Phab, not both.

On Gerrit I wrote:

I not have the capacity to backport Python 3.9 to bullseye nor forward port older Python versions to bullseye, so I proposed adding this for just 3.9, which is missing in CI despite us using it elsewhere.
I hope this is an acceptable solution for needing Python 3.9 CI despite not being perfect.

The CI images have http://apt.wikimedia.org/wikimedia buster-wikimedia thirdparty/pyall and Python 3.8 got added to it via T241195. Adding python 3.9 to it seems the easiest approach and would benefit all the repositories having the tox-docker job.

I mean, if you think it's the easiest option, rather than the patch I wrote which already exists, please, be my guest to backport Python 3.9 to buster. That would be great.

  • migrate the CI tox-docker job to Bullseye, in this case we would need a pyall component with Python 3.5 to 3.9.

This seems like letting perfect be the enemy of the good. Migrating all of tox-docker over to bullseye is explicitly not what is being asked for.

  • for codesearch repository (and any repo that requires python3.9), migrate to Release Pipeline . That also lets one publish an image to our Docker registry and might open the path of migrating code search to k8s.

This seems unreasonable for every single repository we use that has Python. I'm marking this as high priority given that we have production code running on bullseye/Python 3.9, without any CI verifying it.

Change 713972 abandoned by Hashar:

[integration/config@master] Add tox-bullseye container

Reason:

We need either:

- Buster and backport 3.9
- Bullseye and build 3.5, 3.6, 3.7, 3.8 for Bullseye

Last week I have updated the task description T289222

https://gerrit.wikimedia.org/r/713972

See also T270389. 30% of Pywikibot pypi downloads are made for Python 3.9 (T266984) and it would be appropriate to have tests wit this Python version then.

Or drop support for python 3.4, 3.5 and 3.6 entirely and get the new minimum supported version to be 3.7

Why not? All these versions have reached their EOL. There is no syntax change except of the await/async keywords introduced in Python 3.7. Python 3.5 code will run with 3.7+ (except possibly if these keywords are used) whereas Python 3.7 code may not run with 3.5 (e.g. due to ordered dicts, f-strings, circular imports, new methods etc.)