Page MenuHomePhabricator
Create Task
Maniphest T345607

tox 4 no more support [tox:jenkins] configuration section
Closed, ResolvedPublic
Actions

Description

Reported upstream https://github.com/tox-dev/tox/issues/3115

Tox jobs got upgraded to tox 4 (T345152) and as part of the rewrite upstream dropped support for the tox:jenkins section. In the Upstream documentation for the breaking change:

People who still want Jenkins override support may create a tox plugin to achieve this functionality

We typically use tox:jenkins to override the list of environments and/or ensuring to not skip on missing interpreters.

Examples

https://gerrit.wikimedia.org/r/c/labs/tools/wikinity/+/954748

tox.ini
[tox]
envlist = lint,test
skipsdist = True

[tox:jenkins]
envlist = lint,jenkins

[testenv]
setenv = 
  jenkins: FLASK_CONFIG_FILE = wikimedia_config.yaml

[testenv:jenkins]

Under Jenkins tox runs test instead of jenkins and FLASK_CONFIG_FILE environment variable is not set which leads to the failure.

Another example is Quibble which has tox environment using whatever python3 version is available while on Jenkins we run against explicit python versions (3.7, 3.8, 3.9):

tox.ini
[tox]
envlist = check, py3-lint, black, py3-unit, integration

[tox:jenkins]
envlist = check, py{37,38,39}-{lint}, black, py{37,38,39}-{unit}, doc

Details

SubjectRepoBranchLines +/-
dockerfiles: upgrade tox to 4.8.0integration/configmaster+88 -5
dockerfiles: install python3 pip from source v23.2.1integration/configmaster+71 -0
tox v4 support for [tox:jenkins] configurationintegration/tox-jenkins-overridemaster+367 -0
Adjustement for tox 4labs/tools/wikinitymaster+17 -20
Setup tox for integration/tox-jenkins-overrideintegration/configmaster+4 -0
Bring back support for [tox:jenkins]integration/quibblemaster+85 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Sep 5 2023, 9:37 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 5 2023, 9:37 AM
hashar mentioned this in T278203: Migrate all CI jobs from stretch to buster or later and drop stretch testing support.Sep 5 2023, 9:38 AM
hashar added a parent task: T345152: [ci,operations-puppet] upgrade to tox 4 in order to detect changed requirement files.
gerritbot added a comment.Sep 5 2023, 4:56 PM

Change 954971 had a related patch set uploaded (by Hashar; author: Hashar):

[labs/tools/wikinity@master] Adjustement for tox 4

https://gerrit.wikimedia.org/r/954971

gerritbot added a project: Patch-For-Review.Sep 5 2023, 4:56 PM
hashar mentioned this in T345695: Deal with tox 4 upgrading.Sep 6 2023, 8:38 AM
hashar added a subtask: T345695: Deal with tox 4 upgrading.
hashar removed a subtask: T345695: Deal with tox 4 upgrading.
hashar added a parent task: T345695: Deal with tox 4 upgrading.
hashar updated the task description. (Show Details)Sep 6 2023, 8:44 AM
hashar added a comment.Sep 6 2023, 8:48 AM

I gave it a try with a plugin which can be dropped in a toxfile.py in a repo:

toxfile.py
@impl
def tox_add_core_config(core_conf, state):
    if not 'JENKINS_URL' in os.environ:
        return
    core_conf.loaders.insert(0, state.conf._src.get_loader(IniSection('tox', 'jenkins'), {}))

In integration/quibble.git that gives me:

$ tox -l
check
py3-lint
black
py3-unit
integration
$ JENKINS_URL=1 tox -l
check
py37-lint
py38-lint
py39-lint
black
py37-unit
py38-unit
py39-unit
doc

Which looks promising but I dislike using state.conf._src variable which has an underscore prefix indicated it not part of the public API.

gerritbot added a comment.Sep 6 2023, 8:49 AM

Change 954971 merged by jenkins-bot:

[labs/tools/wikinity@master] Adjustement for tox 4

https://gerrit.wikimedia.org/r/954971

gerritbot added a comment.Sep 6 2023, 12:24 PM

Change 955323 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] Bring back support for [tox:jenkins]

https://gerrit.wikimedia.org/r/955323

hashar added a project: Upstream.Sep 6 2023, 1:02 PM
hashar updated the task description. (Show Details)
hashar moved this task from Backlog to Reported Upstream on the Upstream board.
gerritbot added a comment.Sep 6 2023, 6:47 PM

Change 955380 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/tox-jenkins-override@master] tox v4 support for [tox:jenkins] configuration

https://gerrit.wikimedia.org/r/955380

gerritbot added a comment.Sep 6 2023, 8:47 PM

Change 955394 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Setup tox for integration/tox-jenkins-override

https://gerrit.wikimedia.org/r/955394

gerritbot added a comment.Sep 6 2023, 8:48 PM

Change 955394 merged by jenkins-bot:

[integration/config@master] Setup tox for integration/tox-jenkins-override

https://gerrit.wikimedia.org/r/955394

hashar removed a parent task: T345152: [ci,operations-puppet] upgrade to tox 4 in order to detect changed requirement files.Sep 6 2023, 9:12 PM
hashar mentioned this in T345152: [ci,operations-puppet] upgrade to tox 4 in order to detect changed requirement files.
gerritbot added a comment.Sep 7 2023, 1:35 PM

Change 955323 abandoned by Hashar:

[integration/quibble@master] Bring back support for [tox:jenkins]

Reason:

I have made it a plugin in the repository integration/tox-jenkins-override Ife15751da24f28a6479ec7bdd6a902c35f70ff63

The plugin will need to be installed in the Docker base image.

https://gerrit.wikimedia.org/r/955323

gerritbot added a comment.Sep 7 2023, 1:56 PM

Change 955397 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: upgrade tox to 4.8.0

https://gerrit.wikimedia.org/r/955397

gerritbot added a comment.Sep 7 2023, 3:05 PM

Change 955380 merged by jenkins-bot:

[integration/tox-jenkins-override@master] tox v4 support for [tox:jenkins] configuration

https://gerrit.wikimedia.org/r/955380

gerritbot added a comment.Sep 7 2023, 4:13 PM

Change 955780 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: install python3 pip from source v23.2.1

https://gerrit.wikimedia.org/r/955780

gerritbot added a comment.Sep 8 2023, 12:06 PM

Change 955780 merged by jenkins-bot:

[integration/config@master] dockerfiles: install python3 pip from source v23.2.1

https://gerrit.wikimedia.org/r/955780

gerritbot added a comment.Sep 8 2023, 12:12 PM

Change 955919 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: switch tox jobs to pip v23.2.1

https://gerrit.wikimedia.org/r/955919

gerritbot added a comment.Sep 8 2023, 12:12 PM

Change 955920 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: switch zuul-cloner job to pip v23.2.1

https://gerrit.wikimedia.org/r/955920

Stashbot added a comment.Sep 11 2023, 12:24 PM

Mentioned in SAL (#wikimedia-releng) [2023-09-11T12:24:02Z] <hashar> Upgrading pip to 23.2.1 in tox jobs | https://gerrit.wikimedia.org/r/c/integration/config/+/955919 | T345607

Stashbot added a comment.Sep 11 2023, 12:28 PM

Mentioned in SAL (#wikimedia-releng) [2023-09-11T12:28:50Z] <hashar> Rolled back upgrade 23.2.1 for tox jobs | https://gerrit.wikimedia.org/r/c/integration/config/+/955919 | T345607

gerritbot added a comment.Sep 11 2023, 2:09 PM

Change 955397 merged by jenkins-bot:

[integration/config@master] dockerfiles: upgrade tox to 4.8.0

https://gerrit.wikimedia.org/r/955397

Stashbot added a comment.Sep 11 2023, 2:13 PM

Mentioned in SAL (#wikimedia-releng) [2023-09-11T14:13:49Z] <hashar> Building Docker images for upgrading tox to 4.8.0 | https://gerrit.wikimedia.org/r/c/integration/config/+/955397/ | T345695 | T345607

hashar added a comment.Sep 11 2023, 2:47 PM

The releng CI images having tox 4.8.0 and support for tox:jenkins tox configuration section are:

docker-registry.wikimedia.org/releng/tox-mysqld:4.8.0-1
docker-registry.wikimedia.org/releng/tox-buster:4.8.0-1
docker-registry.wikimedia.org/releng/zuul-cloner:0.3.0-s3
docker-registry.wikimedia.org/releng/tox-pyspark:4.8.0-1
docker-registry.wikimedia.org/releng/tox-acme-chief:4.8.0-1
docker-registry.wikimedia.org/releng/tox-java8:4.8.0-1
docker-registry.wikimedia.org/releng/tox-poolcounter:4.8.0-1
docker-registry.wikimedia.org/releng/tox-pywikibot:4.8.0-1
hashar closed this task as Resolved.Sep 11 2023, 2:48 PM
hashar claimed this task.

Solved by the creation of a tox plugin: https://gerrit.wikimedia.org/g/integration/tox-jenkins-override

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL