Page MenuHomePhabricator

tox v4 replacement for TOX_TESTENV_PASSENV
Open, Needs TriagePublic

Description

tox v4 (T345695) has dropped support for TOX_TESTENV_PASSENV which on CI we set to * in order to pass the Jenkins and container environment variables to the test command.

See Upstream discussion at https://github.com/tox-dev/tox/discussions/2681

For CI we have the use case for at least:

XDG_CACHE_HOME: from the above it looks like that can be done by setting TOX_OVERRIDE='testenv.pass_env=XDG_CACHE_HOME

PY_COLORS=1: which I guess can be done using TOX_OVERRIDE='testenv.set_env=PY_COLORS=1.

It looks like it is working:

$ TOX_OVERRIDE=testenv.pass_env=XDG_CACHE_HOME,testenv.set_env=PY_COLORS=1 tox config -k setenv pass_env
[testenv:demo]
set_env = 
  ...
  PY_COLORS=1
pass_env =
  ...
  XDG_CACHE_HOME

The devil is of course whether we rely on other environment variables.

Event Timeline

hashar renamed this task from tox v4 TOX_TESTENV_PASSENV to tox v4 replacement for TOX_TESTENV_PASSENV.Oct 13 2023, 2:47 PM

Change 965750 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] Dump environment within tox env

https://gerrit.wikimedia.org/r/965750

Setting XDG_CACHE_HOME is for the pip cache. For integration/quibble, which has migrated to tox v4, the check testenv complains about a lack of writable cache directory:

check: commands[1]> python setup.py --version
WARNING: The directory '/nonexistent/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
WARNING: The directory '/nonexistent/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
/src/.tox/check/lib/python3.7/site-packages/setuptools/__init__.py:85: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated. Requirements should be satisfied by a PEP 517 installer. If you are using pip, you can try `pip install --use-pep517`.

Once overriding testenv.passenv=XDG_CACHE_HOME:

check: commands[1]> python setup.py --version
/src/.tox/check/lib/python3.7/site-packages/setuptools/__init__.py:85: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated. Requirements should be satisfied by a PEP 517 installer. If you are using pip, you can try `pip install --use-pep517`.
dist.fetch_build_eggs(dist.setup_requires)
0.0.1

Setting PY_COLORS=1 would fix pytest coloring:

pytest.png (319×491 px, 32 KB)

pytest_PY_COLORS=1.png (319×491 px, 32 KB)

Change 965776 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: pass XDG_CACHE_HOME and set PY_COLORS=1

https://gerrit.wikimedia.org/r/965776

Change 965776 merged by jenkins-bot:

[integration/config@master] dockerfiles: pass XDG_CACHE_HOME and set PY_COLORS=1

https://gerrit.wikimedia.org/r/965776

Change 965778 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: switch integration/quibble job to new tox image

https://gerrit.wikimedia.org/r/965778

Change 965778 merged by jenkins-bot:

[integration/config@master] jjb: switch integration/quibble job to new tox image

https://gerrit.wikimedia.org/r/965778

And when dumping the environment variables with /usr/bin/env via https://gerrit.wikimedia.org/r/c/integration/quibble/+/965750 I see:

XDG_CACHE_HOME=/cache
PY_COLORS=1

Which is what I was looking after :)

Change 965750 abandoned by Hashar:

[integration/quibble@master] Dump environment within tox env

Reason:

It works!

https://gerrit.wikimedia.org/r/965750

Change 965783 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: switch tox 4 jobs to new tox image

https://gerrit.wikimedia.org/r/965783

Change 965783 merged by jenkins-bot:

[integration/config@master] jjb: switch tox 4 jobs to new tox image

https://gerrit.wikimedia.org/r/965783

TODO: gotta migrate releng/operations-dnslint and releng/operations-puppet images.

Setting TOX_OVERRIDE=testenv.pass_env=XDG_CACHE_HOME,testenv.set_env=PY_COLORS=1 apparently replaces pass_env and set_env stanzas which might be defined in the sources repositories. At least pywikibot/i18n seems to have been affected: T349599#9308529

Setting TOX_OVERRIDE=testenv.pass_env=XDG_CACHE_HOME,testenv.set_env=PY_COLORS=1 apparently replaces pass_env and set_env stanzas which might be defined in the sources repositories. At least pywikibot/i18n seems to have been affected: T349599#9308529

tox setenv in rPBXQ is also ignored [1] and (PYWIKIBOT_NO_USER_CONFIG is not set) but the test passes anyway (no glue why yet) whereas rPWBC has that tox setenv setting to PYWIKIBOT_NO_USER_CONFIG=2 [2] but tox is 3.21.4 I guess.

[1] https://integration.wikimedia.org/ci/job/pywikibot-xqbot-tox-docker/380/console
[2] https://integration.wikimedia.org/ci/job/pywikibot-core-tox-docker/23572/console

labs/tools/wikinity CI seems to broke because of this, as I made use of pass_env in my tox.ini file. Since the variables I'm accessing (namely, CI) is not listed in Wikimedia CI's TOX_OVERRIDE, tests fully broke. I'm fairly certain this is also why T353075: wikimedia-cz/tracker CI tests fail is happening.

In other words: Using TOX_OVERRIDE on both pass_env and set_env means that user-defined values in tox.ini files are fully ignored. This does not seem warranted – there can always be app-specific environment variables the application needs, and this removes the ability to define them.

Any advice, please?

I went with tox 4.8.0 and further versions have bug fixes related to overriding settings:

4.10.0
--override foo+=bar appending syntax will now work correctly when foo wasn’t defined in tox.ini. (#3100))
4.11.0

Allow passing in multiple overrides using the ; character and fix , being used as splitting values - by @gaborbernat. (#3112)

I guess one can in their repo require a more recent version of tox which should cause tox to create a virtualenv, install a newer version of tox and run it:

tox.ini
[tox]
minversion=4.11.0

I went with tox 4.8.0 cause that is the last supporting Python 3.7:

$ pip3 install tox==4.9.0
ERROR: Ignored the following versions that require a different python version:
 4.10.0 Requires-Python >=3.8;
 4.11.0 Requires-Python >=3.8;
 4.11.1 Requires-Python >=3.8;
 4.11.2 Requires-Python >=3.8;
 4.11.3 Requires-Python >=3.8;
 4.11.4 Requires-Python >=3.8;
 4.9.0 Requires-Python >=3.8
...
ERROR: No matching distribution found for tox==4.9.0

Change 983247 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: tox with Bookworm using pyenv

https://gerrit.wikimedia.org/r/983247

@Urbanecm_WMF @Xqt: I am sorry, tox v4 is broken in that regard :-\

I need CI to:

  • pass XDG_CACHE_HOME which is where pip will store retrieved tarballs or wheels.
  • set PY_COLORS=1 to force some tools to turn color output even when output is not a tty (typically pytest running under jenkins).

In https://gerrit.wikimedia.org/r/c/integration/config/+/965776/1/dockerfiles/tox-buster/run.sh#37 I went to use:

TOX_OVERRIDE=testenv.pass_env=XDG_CACHE_HOME,testenv.set_env=PY_COLORS=1

However, given I went with assignment = that replaces the source repositories [testenv] pass_env and set_env overriding what is set by labs/tools/wikinity:

[testenv]
pass_env = CI

Moreover the override when a repository has a more specific testenv (eg [testenv:py3]) which would erase the given override.

Essentially it is broken :-|

I have send a patch Upstream to always allow PY_COLORS https://github.com/tox-dev/tox/pull/3172

And the other patch for Upstream might be https://github.com/tox-dev/tox/pull/3173 so that when tox processes testenv:whatever, if that has no override it would get the one from testenv which are populated from the tox -x testenv.set_env=PY_COLORS=1.

Change 989504 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: tox v4 image

https://gerrit.wikimedia.org/r/989504

Change #989504 abandoned by Hashar:

[integration/config@master] dockerfiles: tox v4 image

Reason:

https://gerrit.wikimedia.org/r/989504