Page MenuHomePhabricator

Deal with tox 4 upgrading
Open, Needs TriagePublic

Description

Context

T345152 is for the CI job on`operations/puppet` not noticing changes made to the requirements file. The reason is the container image has the requirements pre installed (by running tox --notest when building the image) and tox version 3 was unable to detect changes (tox #149) which is addressed by tox version 4 which is a major rewrite.

I have upgraded the releng/operations-puppet image to use tox (and attached patches to T345152), but being a major rewrite, there are more than a few fixes needed. Some indications are listed at upstream guide Upgrading to tox v4.

«« Code search for tox.ini »»

Some issues I have noticed so far:

  • tox:jenkins is no more supported (T345607)
  • usedevelop=true not working when skipsdist=true T346238
  • tox -e someenv path1 path2 is no more supported. The paths need to be separated by a double dash: tox -e someenv -- path1 path2
  • whitelist_externals is no more recognized and should be replaced by allowlist_externals
  • Python 2.7 is no more recognized since tox delegates the discovery to virtualenv. That has been addressed by pinning virtualenv to 20.21.1.
  • TOX_TESTENV_PASSENV no more supported (discussion 2681), we rely on it to pass environment variable from CI such as XDG_CACHE_HOME.
  • Can't reuse env_dir between environments (issue 425) since each env_dir is tied to the environment name defined in tox. There is a plugin for that: tox-ignore-env-name-mismatch
  • Comments in tox.ini are handled differently, notably it does not recognized an indented comment in a multilne option such as:
tox.ini
[flake8]
extend_ignore =
    # expected 2 blank lines, found 1
    E302

# expected 2 blank lines, found 1 ends up being passed as is. The fix is to deindent it in order for tox to recognize it as a comment :)

This task is to track fallout and adjustments to be made to any affected repositories.

Details

SubjectRepoBranchLines +/-
integration/configmaster+61 -0
integration/configmaster+3 -0
integration/configmaster+63 -1
integration/quibblemaster+35 -3
operations/softwaremaster+1 -1
integration/quibblemaster+11 -6
integration/configmaster+3 -1
pywikibot/coremaster+6 -5
labs/tools/heritagemaster+1 -0
labs/tools/gdrive-to-commonsmaster+1 -1
integration/configmaster+1 -1
integration/configmaster+4 -1 K
labs/tools/map-of-monumentsmaster+6 -6
integration/configmaster+1 -1
integration/configmaster+1 -1
integration/configmaster+1 -1
integration/configmaster+1 -1
integration/configmaster+1 -1
mediawiki/services/poolcountermaster+1 -1
integration/configmaster+1 -1
integration/configmaster+1 -1
integration/configmaster+1 -1
labs/tools/heritagemaster+1 -1
operations/deployment-chartsmaster+1 -1
operations/puppetproduction+1 -1
integration/configmaster+2 -2
integration/configmaster+88 -5
integration/configmaster+1 -1
integration/configmaster+24 -24
integration/configmaster+78 -0
integration/configmaster+2 -2
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 955397 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: upgrade tox to 4.8.0

https://gerrit.wikimedia.org/r/955397

Change 955870 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955870

Change 955871 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/services/poolcounter@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955871

Change 955872 had a related patch set uploaded (by Hashar; author: Hashar):

[labs/tools/heritage@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955872

Change 955874 had a related patch set uploaded (by Hashar; author: Hashar):

[labs/tools/gdrive-to-commons@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955874

Change 955875 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/deployment-charts@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955875

Change 955876 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] envoyproxy: tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955876

Change 955880 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/software@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955880

Change 955870 merged by jenkins-bot:

[integration/config@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955870

Change 955872 merged by jenkins-bot:

[labs/tools/heritage@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955872

Change 956430 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: tox: install virtualenv==20.21.1

https://gerrit.wikimedia.org/r/956430

Change 956430 merged by jenkins-bot:

[integration/config@master] dockerfiles: tox: install virtualenv==20.21.1

https://gerrit.wikimedia.org/r/956430

Change 955919 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: tox jobs to pip v23.2.1 virtualenv 20.21.1

https://gerrit.wikimedia.org/r/955919

Change 955920 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: zuul-cloner job to pip v23.2.1 virtualenv 20.21.1

https://gerrit.wikimedia.org/r/955920

Change 955871 merged by jenkins-bot:

[mediawiki/services/poolcounter@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955871

Mentioned in SAL (#wikimedia-releng) [2023-09-11T13:43:46Z] <hashar> Upgrading tox jobs to pip 23.2.1 and virtualenv 20.21.1 | T345695

Change 955919 merged by jenkins-bot:

[integration/config@master] jjb: tox jobs to pip v23.2.1 virtualenv 20.21.1

https://gerrit.wikimedia.org/r/955919

Change 955920 merged by jenkins-bot:

[integration/config@master] jjb: zuul-cloner job to pip v23.2.1 virtualenv 20.21.1

https://gerrit.wikimedia.org/r/955920

Change 955397 merged by jenkins-bot:

[integration/config@master] dockerfiles: upgrade tox to 4.8.0

https://gerrit.wikimedia.org/r/955397

The releng CI images having tox 4.8.0 are:

docker-registry.wikimedia.org/releng/tox-mysqld:4.8.0-1
docker-registry.wikimedia.org/releng/tox-buster:4.8.0-1
docker-registry.wikimedia.org/releng/zuul-cloner:0.3.0-s3
docker-registry.wikimedia.org/releng/tox-pyspark:4.8.0-1
docker-registry.wikimedia.org/releng/tox-acme-chief:4.8.0-1
docker-registry.wikimedia.org/releng/tox-java8:4.8.0-1
docker-registry.wikimedia.org/releng/tox-poolcounter:4.8.0-1
docker-registry.wikimedia.org/releng/tox-pywikibot:4.8.0-1

Change 956453 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update integration/config to tox 4.8.0

https://gerrit.wikimedia.org/r/956453

Change 956453 merged by jenkins-bot:

[integration/config@master] jjb: update integration/config to tox 4.8.0

https://gerrit.wikimedia.org/r/956453

Change 955876 merged by JMeybohm:

[operations/puppet@production] envoyproxy: tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955876

Change 955875 merged by jenkins-bot:

[operations/deployment-charts@master] update_version: tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955875

Change 956795 had a related patch set uploaded (by Hashar; author: Hashar):

[labs/tools/heritage@master] tox.ini: add jsonschema_cmd.sh to allowlist_externals

https://gerrit.wikimedia.org/r/956795

Change 956803 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: upgrade labs/tools/heritage to tox v4

https://gerrit.wikimedia.org/r/956803

Change 956803 merged by jenkins-bot:

[integration/config@master] jjb: upgrade labs/tools/heritage to tox v4

https://gerrit.wikimedia.org/r/956803

Change 956827 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: upgrade labs/tools/map-of-monuments to tox v4

https://gerrit.wikimedia.org/r/956827

Change 956828 had a related patch set uploaded (by Hashar; author: Hashar):

[labs/tools/map-of-monuments@master] tox.ini: adjust flake8 config for tox v4

https://gerrit.wikimedia.org/r/956828

Change 956827 merged by jenkins-bot:

[integration/config@master] jjb: upgrade labs/tools/map-of-monuments to tox v4

https://gerrit.wikimedia.org/r/956827

Change 956870 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: upgrade labs/tools/wikinity to tox v4

https://gerrit.wikimedia.org/r/956870

Change 956870 merged by jenkins-bot:

[integration/config@master] jjb: upgrade labs/tools/wikinity to tox v4

https://gerrit.wikimedia.org/r/956870

Change 956871 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: upgrade mediawiki/services/poolcounter to tox v4

https://gerrit.wikimedia.org/r/956871

Change 956871 merged by jenkins-bot:

[integration/config@master] jjb: upgrade mediawiki/services/poolcounter to tox v4

https://gerrit.wikimedia.org/r/956871

Change 956872 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: zuul-cloner job with tox v4

https://gerrit.wikimedia.org/r/956872

Change 956872 merged by jenkins-bot:

[integration/config@master] jjb: zuul-cloner job with tox v4

https://gerrit.wikimedia.org/r/956872

Change 956874 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: upgrade wikimedia-cz/tracker job to tox v4

https://gerrit.wikimedia.org/r/956874

Change 956874 merged by jenkins-bot:

[integration/config@master] jjb: upgrade wikimedia-cz/tracker job to tox v4

https://gerrit.wikimedia.org/r/956874

Change 956879 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: upgrade cergen to tox v4

https://gerrit.wikimedia.org/r/956879

Change 956880 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: upgrade operations/software/acme-chief to tox v4

https://gerrit.wikimedia.org/r/956880

Change 956879 merged by jenkins-bot:

[integration/config@master] jjb: upgrade cergen to tox v4

https://gerrit.wikimedia.org/r/956879

Change 956880 merged by jenkins-bot:

[integration/config@master] jjb: upgrade operations/software/acme-chief to tox v4

https://gerrit.wikimedia.org/r/956880

Change 956891 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: upgrade tox-poetry-docker job to tox v4

https://gerrit.wikimedia.org/r/956891

Change 956828 merged by jenkins-bot:

[labs/tools/map-of-monuments@master] tox.ini: adjust flake8 config for tox v4

https://gerrit.wikimedia.org/r/956828

Change 956891 merged by jenkins-bot:

[integration/config@master] Remove support for tox + Poetry

https://gerrit.wikimedia.org/r/956891

Change 957277 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: upgrade pywikibot repos to tox 4.8.0

https://gerrit.wikimedia.org/r/957277

Change 957277 merged by jenkins-bot:

[integration/config@master] jjb: upgrade pywikibot i18 and xqbot tox 4.8.0

https://gerrit.wikimedia.org/r/957277

Change 955874 merged by jenkins-bot:

[labs/tools/gdrive-to-commons@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955874

Change 956795 merged by jenkins-bot:

[labs/tools/heritage@master] tox.ini: add jsonschema_cmd.sh to allowlist_externals

https://gerrit.wikimedia.org/r/956795

Change 960006 had a related patch set uploaded (by Hashar; author: Hashar):

[pywikibot/core@master] [test] move doctest_skip to pytest command

https://gerrit.wikimedia.org/r/960006

Change 960006 merged by jenkins-bot:

[pywikibot/core@master] [test] move doctest_skip to pytest command

https://gerrit.wikimedia.org/r/960006

Change 960082 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] Require tox v4 and remove skipsdist/use_develop

https://gerrit.wikimedia.org/r/960082

Change 960091 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: switch integration/quibble tox jobs to tox v4

https://gerrit.wikimedia.org/r/960091

Change 960091 merged by jenkins-bot:

[integration/config@master] jjb: switch integration/quibble tox jobs to tox v4

https://gerrit.wikimedia.org/r/960091

Change 960082 merged by jenkins-bot:

[integration/quibble@master] Require tox v4 and remove skipsdist/use_develop

https://gerrit.wikimedia.org/r/960082

Another one I have found is TOX_TESTENV_PASSENV is no more supported. The tox image sets it to * in order to pass all the CI environment variable (from Jenkins or from the image such as XDG_CACHE_HOME and PY_COLORS=1). https://github.com/tox-dev/tox/discussions/2681.

That can potentially be worked around using TOX_OVERRIDE=testenv.passenv=XDG_CACHE_HOME.

Change 955880 merged by Elukey:

[operations/software@master] tox.ini: whitelist_externals -> allowlist_externals

https://gerrit.wikimedia.org/r/955880

Change 982641 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] release: Quibble 1.6.0

https://gerrit.wikimedia.org/r/982641

Change 982641 merged by jenkins-bot:

[integration/quibble@master] release: Quibble 1.6.0

https://gerrit.wikimedia.org/r/982641

Change 982814 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: update Quibble to 1.6.0

https://gerrit.wikimedia.org/r/982814

Change 982814 merged by jenkins-bot:

[integration/config@master] dockerfiles: update Quibble to 1.6.0

https://gerrit.wikimedia.org/r/982814

Change 989504 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: tox v4 image

https://gerrit.wikimedia.org/r/989504

It turns out tox v4 has a lot of breaking changes and I am not sure whether it is worth pursuing further. The OpenDev community has migrated some of their repositories to use Nox (https://nox.thea.codes/en/stable/index.html). Some discussion they had a year or so ago on that topic: https://meetings.opendev.org/meetings/infra/2023/infra.2023-01-10-19.01.log.html#l-76

Change #1014040 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] tox.ini: require virtualenv<20.22.0 for Python 2.7

https://gerrit.wikimedia.org/r/1014040

Change #1014040 merged by jenkins-bot:

[integration/config@master] tox.ini: require virtualenv<20.22.0 for Python 2.7

https://gerrit.wikimedia.org/r/1014040

Change #989504 abandoned by Hashar:

[integration/config@master] dockerfiles: tox v4 image

Reason:

https://gerrit.wikimedia.org/r/989504