CI: upgrade tox, currently running 2.6.0
Closed, ResolvedPublic

Description

CI is running a pretty updated tox at the moment 2.6.0 and as such has some unfixed bugs and missing features.

It would be nice to run a more modern version of it, like 2.9.1, if we don't want to go directly to the 3.0.0.

The changelog is at https://tox.readthedocs.io/en/latest/changelog.html

Volans created this task.Jun 7 2018, 11:36 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2018, 11:36 AM
Vvjjkkii renamed this task from CI: upgrade tox, currently running 2.6.0 to fhbaaaaaaa.Jul 1 2018, 1:05 AM
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot renamed this task from fhbaaaaaaa to CI: upgrade tox, currently running 2.6.0.
CommunityTechBot added a subscriber: Aklapper.

This is overdue and should be done. I eventually had the use case for -q (--quiet) which got introduced in version 3.0.0.

hashar updated the task description. (Show Details)Jan 10 2019, 4:28 PM

Change 483464 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: bump tox 2.6.0..2.9.1

https://gerrit.wikimedia.org/r/483464

hashar claimed this task.Jan 10 2019, 4:42 PM
hashar triaged this task as Low priority.
hashar moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.

Change 483464 merged by jenkins-bot:
[integration/config@master] docker: bump tox 2.6.0..2.9.1

https://gerrit.wikimedia.org/r/483464

Rebuilding:

[contint1001.wikimedia.org] out: * docker-registry.discovery.wmnet/releng/tox:0.3.0
[contint1001.wikimedia.org] out: * docker-registry.discovery.wmnet/releng/tox-mysqld:0.3.0
[contint1001.wikimedia.org] out: * docker-registry.discovery.wmnet/releng/tox-cergen:0.3.0
[contint1001.wikimedia.org] out: * docker-registry.discovery.wmnet/releng/tox-labs-striker:0.3.0
[contint1001.wikimedia.org] out: * docker-registry.discovery.wmnet/releng/tox-poolcounter:0.3.0
[contint1001.wikimedia.org] out: * docker-registry.discovery.wmnet/releng/tox-pyspark:0.3.0
[contint1001.wikimedia.org] out: * docker-registry.discovery.wmnet/releng/tox-pywikibot:0.3.0
[contint1001.wikimedia.org] out: * docker-registry.discovery.wmnet/releng/tox-certcentral:0.3.0
[contint1001.wikimedia.org] out: * docker-registry.discovery.wmnet/releng/tox-conftool:0.3.0
[contint1001.wikimedia.org] out: * docker-registry.discovery.wmnet/releng/tox-eventlogging:0.3.0

Change 484200 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: fix tox-pyspark gpg under stretch

https://gerrit.wikimedia.org/r/484200

Change 484200 merged by Hashar:
[integration/config@master] docker: fix tox-pyspark gpg under stretch

https://gerrit.wikimedia.org/r/484200

[contint1001.wikimedia.org] out: ERROR: image docker-registry.discovery.wmnet/releng/tox-pyspark failed to build, see logs for details. The generated Dockerfile is:

Dockerfile
FROM docker-registry.discovery.wmnet/releng/tox:0.3.0

USER root

COPY KEYS /tmp/KEYS
COPY spark-2.1.2-bin-hadoop2.6.tgz.asc /tmp/spark-2.1.2-bin-hadoop2.6.tgz.asc

# liblapack3 and libgomp1 are specifically needed for search/MjoLniR which
# runs numerical algorithms
RUN echo 'Acquire::http::Proxy "http://webproxy.eqiad.wmnet:8080";' > /etc/apt/apt.conf.d/80_proxy \
    && apt-get update  \
    && DEBIAN_FRONTEND=noninteractive \
    apt-get install  --yes openjdk-7-jre-headless maven liblapack3 libgomp1 wget --no-install-recommends \
    && rm -f /etc/apt/apt.conf.d/80_proxy \
    && apt-get clean && rm -rf /var/lib/apt/lists/*  && \
    cd /tmp && \
    wget http://archive.apache.org/dist/spark/spark-2.1.2/spark-2.1.2-bin-hadoop2.6.tgz && \
    gpg --import /tmp/KEYS && \
    gpg --verify spark-2.1.2-bin-hadoop2.6.tgz.asc && \
    tar -C /opt -zxf spark-2.1.2-bin-hadoop2.6.tgz && \
    rm spark-2.1.2-bin-hadoop2.6.tgz* KEYS && \
    apt-get purge --yes wget && \
    rm -rf ~/.gnupg

COPY log4j.properties /opt/spark-2.1.2-bin-hadoop2.6/conf/log4j.properties
COPY maven-settings.xml /etc/maven/settings.xml

USER nobody
ENV SPARK_HOME /opt/spark-2.1.2-bin-hadoop2.6

Change 484202 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: tox-pyspark Java 7 -> 8

https://gerrit.wikimedia.org/r/484202

Change 484202 merged by Hashar:
[integration/config@master] docker: tox-pyspark Java 7 -> 8

https://gerrit.wikimedia.org/r/484202

Change 484206 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Update integration tox jobs to releng/tox:0.3.0

https://gerrit.wikimedia.org/r/484206

Change 484207 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Update pywikibot jobs to releng/tox-pywikibot:0.3.0

https://gerrit.wikimedia.org/r/484207

Change 484209 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Update search/mjolnir tox job to 0.3.0 container

https://gerrit.wikimedia.org/r/484209

Change 484206 merged by jenkins-bot:
[integration/config@master] Update integration tox jobs to releng/tox:0.3.0

https://gerrit.wikimedia.org/r/484206

Change 484207 merged by jenkins-bot:
[integration/config@master] Update pywikibot jobs to releng/tox-pywikibot:0.3.0

https://gerrit.wikimedia.org/r/484207

Change 484209 merged by jenkins-bot:
[integration/config@master] Update search/mjolnir tox job to 0.3.0 container

https://gerrit.wikimedia.org/r/484209

I have upgraded all containers/jobs to containers releng/toxXXXXX:0.3.0. That comes with tox 2.9.1 and the containers also have multiple python versions (from T191764).

@hashar thanks a lot, CI is finally running successfully for Debmonitor (i.e. https://gerrit.wikimedia.org/r/c/operations/software/debmonitor/+/483131 )

Cool! Thanks for the confirmation.

The new container also bump from Jessie to Stretch AND has several python3 versions available :-]

hashar closed this task as Resolved.Jan 14 2019, 5:58 PM