Page MenuHomePhabricator
Create Task
Maniphest T222512

CI: upgrade tox or allow to override its version per-project
Closed, ResolvedPublic
Actions

Description

We have tox 2.9.1 (released on 2017-09-29) in CI right now and there have been 23 tox releases since then, including 1 major and 9 minor ones.

It would be nice to have a more modern one in CI.
I totally understand that upgrading tox in CI is delicate as it affects all the repositories that uses it.
An alternative approach would be to allow each project to optionally override the default tox version to require a newer one.

For example in https://gerrit.wikimedia.org/r/c/operations/software/cumin/+/508079/ I had to use a newer syntax to simply a lot the tox.ini configuration but that syntax was added in 3.0.0 and of course is failing CI.

Details

SubjectRepoBranchLines +/-
Bump misc tox jobs to tox 3.10.0integration/configmaster+14 -14
docker: rebuild based on releng/tox:0.4.0integration/configmaster+63 -0
Bump most jobs to tox 3.10.0integration/configmaster+8 -8
Switch cumin to tox 3.10.0integration/configmaster+2 -2
Bump most jobs to tox 3.10.0integration/configmaster+8 -8
Switch cumin to tox 3.10.0integration/configmaster+2 -0
Allow image override in tox publish templatesintegration/configmaster+4 -2
docker: bump tox from 2.9.1 to 3.10.0integration/configmaster+7 -1
Customize query in gerrit

Related Objects

Event Timeline

Volans created this task.May 4 2019, 2:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 4 2019, 2:32 PM
hashar added a comment.May 13 2019, 1:11 PM

With the current CI we do not have a good way to use a different tox version per repository, the upgrade is for all repositories :-/ Though lot of tox jobs are per repository so they could be made to use a different container.

https://tox.readthedocs.io/en/latest/changelog.html

gerritbot added a comment.May 13 2019, 1:13 PM

Change 509824 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: bump tox from 2.9.1 to 3.10.0

https://gerrit.wikimedia.org/r/509824

gerritbot added a project: Patch-For-Review.May 13 2019, 1:13 PM
hashar claimed this task.May 13 2019, 1:24 PM
Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptMay 13 2019, 1:24 PM
gerritbot added a comment.May 13 2019, 1:25 PM

Change 509824 merged by jenkins-bot:
[integration/config@master] docker: bump tox from 2.9.1 to 3.10.0

https://gerrit.wikimedia.org/r/509824

gerritbot added a comment.May 13 2019, 1:29 PM

Change 509839 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Allow image override in tox publish templates

https://gerrit.wikimedia.org/r/509839

gerritbot added a comment.May 13 2019, 1:29 PM

Change 509840 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Switch cumin to tox 3.10.0

https://gerrit.wikimedia.org/r/509840

hashar added a comment.May 13 2019, 1:35 PM

That is done for operations/software/cumin.

hashar triaged this task as Medium priority.May 13 2019, 1:35 PM
hashar moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
hashar moved this task from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.
gerritbot added a comment.May 13 2019, 1:37 PM

Change 509839 merged by jenkins-bot:
[integration/config@master] Allow image override in tox publish templates

https://gerrit.wikimedia.org/r/509839

gerritbot added a comment.May 13 2019, 1:37 PM

Change 509840 merged by jenkins-bot:
[integration/config@master] Switch cumin to tox 3.10.0

https://gerrit.wikimedia.org/r/509840

Volans added a comment.May 13 2019, 1:45 PM

@hashar thanks for the patch, I'm getting this error though:

13:43:33 Unable to find image 'docker-registry.wikimedia.org/releng/tox:0.4.0' locally
13:43:33 docker: Error response from daemon: manifest for docker-registry.wikimedia.org/releng/tox:0.4.0 not found.

See https://integration.wikimedia.org/ci/job/cumin-tox-docker/177/console

gerritbot added a comment.May 13 2019, 1:47 PM

Change 509849 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Bump most jobs to tox 3.10.0

https://gerrit.wikimedia.org/r/509849

hashar added a comment.May 13 2019, 2:15 PM

Eventually the new image docker-registry.wikimedia.org/releng/tox:0.4.0 can not be found on the registry for some reason :-(

gerritbot added a comment.May 13 2019, 2:19 PM

Change 509849 merged by jenkins-bot:
[integration/config@master] Bump most jobs to tox 3.10.0

https://gerrit.wikimedia.org/r/509849

hashar added a comment.May 13 2019, 2:28 PM

Fabian explained it is a replication issue in the docker registry. So just have to wait (tm).

I have reverted the jobs. Will redeploy them later once I can confirm the image can properly be pulled, eg on integration-cumin:

sudo cumin --force 'name:docker' 'docker pull docker-registry.wikimedia.org/releng/tox:0.4.0'
hashar changed the task status from Open to Stalled.May 13 2019, 2:32 PM

Pending availability of the image T222210#5176863

gerritbot added a comment.May 14 2019, 8:41 AM

Change 510086 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Switch cumin to tox 3.10.0

https://gerrit.wikimedia.org/r/510086

hashar changed the task status from Stalled to Open.May 14 2019, 8:41 AM

docker-registry.wikimedia.org/releng/tox:0.4.0 is now available on all Jenkins instances.

gerritbot added a comment.May 14 2019, 8:47 AM

Change 510087 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Bump most jobs to tox 3.10.0

https://gerrit.wikimedia.org/r/510087

gerritbot added a comment.May 14 2019, 8:48 AM

Change 510086 merged by jenkins-bot:
[integration/config@master] Switch cumin to tox 3.10.0

https://gerrit.wikimedia.org/r/510086

gerritbot added a comment.May 14 2019, 8:51 AM

Change 510087 merged by jenkins-bot:
[integration/config@master] Bump most jobs to tox 3.10.0

https://gerrit.wikimedia.org/r/510087

gerritbot added a comment.May 14 2019, 9:11 AM

Change 510091 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: rebuild based on releng/tox:0.4.0

https://gerrit.wikimedia.org/r/510091

gerritbot added a comment.May 14 2019, 9:18 AM

Change 510091 merged by jenkins-bot:
[integration/config@master] docker: rebuild based on releng/tox:0.4.0

https://gerrit.wikimedia.org/r/510091

gerritbot added a comment.May 14 2019, 9:31 AM

Change 510096 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Bump misc tox jobs to tox 3.10.0

https://gerrit.wikimedia.org/r/510096

hashar closed this task as Resolved.May 14 2019, 9:43 AM

All containers have been updated except the special cases:

  • operations-dnslint : uses the latest tox version available at the time of rebuild
  • operations-puppet : pinned to 1.9.2 for historical reasons (but could probably be upgraded)
gerritbot added a comment.May 14 2019, 9:45 AM

Change 510096 merged by jenkins-bot:
[integration/config@master] Bump misc tox jobs to tox 3.10.0

https://gerrit.wikimedia.org/r/510096

hashar added a comment.May 14 2019, 9:48 AM

Announced on the QA list.

JeanFred mentioned this in T231152: Flake8 fails on labs-tools-heritage, complaining about py3 stuff.Aug 25 2019, 8:42 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 25 2019, 9:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits