Change 668176 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Work around python2 in tox container

https://gerrit.wikimedia.org/r/668176

I am not sure what leads to:

/usr/bin/python -m virtualenv --no-download --python /usr/bin/python venv-py3 >venv-py3/log/check-0.log

Even if tox runs with python2, it parses tox.ini and lookup on disk to find the appropriate python executable to use (for example python3.6) and should then pass that to virtualenv --python XXX which would craft the venv with the appropriate version.

I did a bunch of mess when adding support for multiple python versions with https://gerrit.wikimedia.org/r/c/integration/quibble/+/607512 , the two commands I have used to verify were:

tox --listenvs-all --showconfig|egrep '^(\[|envdir|basepython)'
JENKINS_URL=1 tox --showconfig|egrep '^(\[|envdir|basepython)'

But maybe I messed up something since my tox was probably installed with python3.

The base image installs tox with pip which is the python2.7 version of pip. So maybe it ends up defaulting to python 2.7 in some cases. We can try the above tox --show-config commands in the tox-buster image and see what happens? If anything lists basepython = python2.7 or basepython = python we indeed a configuration issue in tox.

There is also some discrepancy which is that I probably based my work on the releng/tox image which had tox 3.10.0. There is now a tox-buster image which had tox upgraded to 3.21.4 ( T274232) which might be related.

Anyway yeah that is a bit of rabbit hole to dig in.

Do you have link to something that suddenly started failing? Is that a ci build failure that is unrelated to a change you proposed? In this case that would hint at either tox-buster or tox 3.21.4 being the culprits.

To clarify: I pretty sure it is a behavior change introduced by the tox 3.10.0 to 3.21.4

The CI tooling being hit by a regression in the CI stack is very very meta. I can't really investigate deeply into it tonight though.

Sorry, I left out the error report itself, I was seeing unrelated CI failures from my quibble patches, for example this run which fails like,

00:00:18.628 using tox-3.21.4 from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc (pid 42)
00:00:18.629 GLOB sdist-make: /src/setup.py
00:00:18.635 [58] /src$ /usr/bin/python setup.py sdist --formats=zip --dist-dir /src/.tox/dist >.tox/log/GLOB-0.log
00:00:20.959 package .tmp/package/1/quibble-0.1.dev2+g0c9950a.zip links to dist/quibble-0.1.dev2+g0c9950a.zip (/src/.tox)
00:00:20.994 check cannot reuse: no previous config /src/.tox/venv-py3/.tox-config1
00:00:20.994 check create: /src/.tox/venv-py3
00:00:21.001 [77] /src/.tox$ /usr/bin/python -m virtualenv --no-download --python /usr/bin/python venv-py3 >venv-py3/log/check-0.log
00:00:21.815 check installdeps: .[test]
00:00:21.821 [94] /src$ /src/.tox/venv-py3/bin/python -m pip install '.[test]' >.tox/venv-py3/log/check-1.log
00:00:30.913 ERROR: invocation failed (exit code 1), logfile: /src/.tox/venv-py3/log/check-1.log
00:00:30.913 ================================== log start ===================================
00:00:30.914 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
00:00:30.914 Processing /src
00:00:30.914   Installing build dependencies: started
00:00:30.914   Installing build dependencies: finished with status 'done'
00:00:30.914   Getting requirements to build wheel: started
00:00:30.914   Getting requirements to build wheel: finished with status 'done'
00:00:30.914   Installing backend dependencies: started
00:00:30.914   Installing backend dependencies: finished with status 'done'
00:00:30.914     Preparing wheel metadata: started
00:00:30.914     Preparing wheel metadata: finished with status 'done'
00:00:30.914 ERROR: Package u'quibble' requires a different Python: 2.7.16 not in '>=3.5'
00:00:30.914 
00:00:30.914 =================================== log end ====================================

We're trying to pip install with /src/.tox/venv-py3/bin/python, and it's py27. It seems worthwhile regardless to remove py2 from all tox* CI images, it's long-since unsupported. Especially if it's suspected of causing problems!

The workaround patch attached to this task seems to demonstrate that forcing py3 everywhere successfully avoids the root cause...

Yes forcing python3 is definitely the thing to do. However when one does:

[testenv]
basepython = python3

tox will enforces python3 everywhere even if a factor name implies another version (ex: py32), though it would emit a warning about the version conflict if python3 is not python 3.2 implied by the factor py32.

The solution is to instruct tox to ignore the conflict which as a side effect also force set the python version from the factor name. That is https://tox.readthedocs.io/en/latest/config.html#conf-ignore_basepython_conflict The option is poorly named, it ignore the warning but ENFORCES the proper version to be used.

So we need:

[tox]
# Enforce the python version determined from the factor (ex: py37)
# effectively overwriting the default basepython below.
ignore_basepython_conflict = True

[testenv]
# Default python to use for environment names not defining a python version (ex: doc).
basepython = python3

I gave it a try with tox 3.21.4 installed with python2:

$ head -n1 $(which tox)
#!/usr/bin/python

$ python --version
Python 2.7.16

$ tox --version
3.21.4 imported from lib/python2.7/site-packages/tox/__init__.pyc

Debug command:

$ JENKINS_URL=1 tox --showconfig|egrep '(host_python|^tox =|^\[|basepython)'
[tox]
ignore_basepython_conflict = True
host_python = /usr/bin/python
[tox:versions]
tox = 3.21.4
[testenv:check]
basepython = python3
[testenv:py35-lint]
basepython = python3.5
[testenv:py36-lint]
basepython = python3.6
[testenv:py37-lint]
basepython = python3.7
[testenv:py38-lint]
basepython = python3.8
[testenv:black]
basepython = python3
[testenv:py35-unit]
basepython = python3.5
[testenv:py36-unit]
basepython = python3.6
[testenv:py37-unit]
basepython = python3.7
[testenv:py38-unit]
basepython = python3.8
[testenv:doc]
basepython = python3
[testenv:cover]
basepython = python3
[testenv:venv]
basepython = python3

Which shows that python3 is picked up (due to basepython = python3 and for the pyXX-{lint,unit} environment the proper python3.x is used (as implied by their name and FORCED by ignore_basepython_conflict).

Really too late to figure out the root cause. Maybe it was already broken and further broken by the tox upgrade but the above would fix it.

11pm there so I am falling asleep but we can finish that tomorrow :] And I guess write yet another blog post!

Change 668342 had a related patch set uploaded (by Awight; owner: Awight):
[integration/config@master] Don't include python2 in the tox-buster image

https://gerrit.wikimedia.org/r/668342

Thanks for finding the fix! Latest patch includes your changes, and local tox-buster:0.3.0 tests succeed.

Change 668176 abandoned by Awight:
[integration/quibble@master] Work around python2 in tox container

Reason:
You were right: when tox is installed using py3, that becomes the default version with no extra configuration required.

https://gerrit.wikimedia.org/r/668176

The ignore_basepython_conflict setting I found out yesterday night is a great hack, then it is really just a hack an any repository might encounter the exact same issue (using python2.7 as the basepython) which would need one to do the same analysis I have done, reach out to the same conclusion and add some more cryptic config to their tox.ini.

I would imagine that instead we can just have the CI image to install tox with pip3. This way it would default to python3 instead of python (which is 2.7) and that saves us from having to do dirty hacks on all repositories. The only drawback is that it might break repositories that relies on python 2.7 being the default but we can afford to break those and they can just basepython = python2.7. The last majority of repositories will be better with a default of python3.

I think the fix is to replace pip install by pip3 install in the releng/tox-buster image:

dockerfiles/tox-buster/Dockerfile.template:32:    && pip install setuptools \
dockerfiles/tox-buster/Dockerfile.template:33:    && pip install tox==3.21.4 \

That would make tox to create the virtualenv with /usr/bin/python3 by default and will be sane for most repositories.

We leave the legacy releng/tox image as is (it is using tox==3.10.0) but that image will eventually be removed as all jobs get migrated to releng/tox-buster.

Change 668342 merged by jenkins-bot:
[integration/config@master] Install tox with python3

https://gerrit.wikimedia.org/r/668342

Successfully published image docker-registry.discovery.wmnet/releng/tox-pywikibot:0.7.0
Successfully published image docker-registry.discovery.wmnet/releng/tox-buster:0.4.0
Successfully published image docker-registry.discovery.wmnet/releng/tox-censorshipmonitoring:0.3.0
Successfully published image docker-registry.discovery.wmnet/releng/tox-conftool:0.6.0
Successfully published image docker-registry.discovery.wmnet/releng/tox-labs-striker:0.7.0
Successfully published image docker-registry.discovery.wmnet/releng/tox-homer:0.3.0
Successfully published image docker-registry.discovery.wmnet/releng/tox-eventlogging:0.6.0

Change 668424 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: use python3 tox for quibble repository

https://gerrit.wikimedia.org/r/668424

Change 668424 merged by jenkins-bot:
[integration/config@master] jjb: use python3 tox for quibble repository

https://gerrit.wikimedia.org/r/668424

Change 668467 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: use a python3 based tox

https://gerrit.wikimedia.org/r/668467

Mentioned in SAL (#wikimedia-releng) [2021-03-04T15:47:03Z] <hashar> Refreshing jobs based on releng/tox-buster to use latest image. That brings in tox installed with python3 instead of python2 # T276384

Change 668467 merged by jenkins-bot:
[integration/config@master] jjb: use a python3 based tox

https://gerrit.wikimedia.org/r/668467