Spicerack's tox config times out all the time after T342019
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• elukey
	Aug 14 2024, 4:20 PM

Project Tags

Referenced Files

None

Subscribers

Description

Right after T342019 the Spicerack's tox config forces CI to run all the supported Python versions, in our case all the ones listed in the tox.ini (3.9, 3.10, 3.11). This takes a ton of time, and right around py311 it times out (20 mins hard stop).

Example: https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1060854

The only viable option in my opinion is to modify our CI config to just run tox -e pyXX, with XX equal to the Python version that we run on cumin nodes. More tox environments will be available locally (so we can test for newer versions), but we cannot really wait minutes and minutes for CI in my opinion.

@hashar any suggestion on how to implement it?

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
tox: only install flake8 when running flake8	Hashar	operations/software/spicerack	master	+5 -2
tox: run less environments on CI	Hashar	operations/software/spicerack	master	+8 -0
tox: add config for jenkins	Elukey	operations/software/spicerack	master	+5 -0

Customize query in gerrit

Related Objects

Mentioned In: T365372: Spicerack: expand Supermicro support in the Redfish module
Mentioned Here: T342019: Add Python 3.10, 3.11 and 3.12 to Wikimedia CI

Event Timeline

• elukey created this task.Aug 14 2024, 4:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 14 2024, 4:20 PM

• elukey added a project: Release-Engineering-Team.Aug 14 2024, 4:23 PM

• elukey updated the task description. (Show Details)

• elukey added a subscriber: hashar.

• elukey added a project: User-Elukey.Aug 14 2024, 4:26 PM

• elukey updated the task description. (Show Details)

• elukey moved this task from Backlog to In Progress on the User-Elukey board.

• ayounsi subscribed.Aug 16 2024, 9:23 AM

• elukey mentioned this in T365372: Spicerack: expand Supermicro support in the Redfish module.Aug 16 2024, 4:29 PM

• elukey triaged this task as High priority.Aug 26 2024, 2:42 PM

T342019 added Python 3.10 and 3.11 to the image which indeed caused our Jenkins CI to run all combinations defined in tox.ini:

$ tox -l
py39-flake8
py39-unit
py39-bandit
py39-mypy
py39-prospector
py39-sphinx
py310-flake8
py310-unit
py310-bandit
py310-mypy
py310-prospector
py310-sphinx
py311-flake8
py311-unit
py311-bandit
py311-mypy
py311-prospector
py311-sphinx
py39-style
py310-style
py311-style

So that each of flake8 → unit → bandit → mypy → prospector → sphinx → style is run against each of the pythons (3.9 → 3.10 → 3.11) which is probably redundant.

Looking at a timing out build, the py39-flake8 tooks 45 seconds to install the dependencies when it runs in 1.2 seconds (timestamps are relative to the start of the Jenkins build):

00:00:08.294 py39-flake8 create: /src/.tox/py39-flake8
00:00:12.809 py39-flake8 installdeps: .[tests]
00:00:48.058 py39-flake8 develop-inst: /src
00:00:57.643 py39-flake8 installed: alabaster==0.7.16,attrs==24.2.0,babel==2.16.0,bandit==1.7.9,cachetools==5.4.0,certifi==2024.7.4,cffi==1.17.0,charset-normalizer==3.3.2,ClusterShell==1.9.2,conftool==3.2.1,coverage==7.6.1,cryptography==43.0.0,cumin==4.2.0,Deprecated==1.2.14,dnspython==2.0.0,docutils==0.20.1,elasticsearch==7.14.2,exceptiongroup==1.2.2,execnet==2.1.1,flake8==7.1.1,gitdb==4.0.11,GitPython==3.1.43,google-auth==2.33.0,idna==3.7,imagesize==1.4.1,importlib_metadata==8.2.0,iniconfig==2.0.0,Jinja2==3.1.4,jsonschema==4.23.0,jsonschema-specifications==2023.12.1,kafka-python==2.0.2,kubernetes==12.0.1,markdown-it-py==3.0.0,MarkupSafe==2.1.5,mccabe==0.7.0,mdurl==0.1.2,mypy==1.11.1,mypy-extensions==1.0.0,oauthlib==3.2.2,packaging==24.1,pbr==6.0.0,phabricator==0.9.1,pluggy==1.5.0,pyasn1==0.6.0,pyasn1_modules==0.4.0,pycodestyle==2.12.1,pycparser==2.22,pyflakes==3.2.0,Pygments==2.18.0,PyMySQL==1.1.1,pynetbox==6.6.2,pyparsing==3.1.2,pytest==8.3.2,pytest-cov==5.0.0,pytest-xdist==3.6.1,python-dateutil==2.9.0.post0,python-etcd==0.4.5,PyYAML==6.0.2,redis==4.1.3,referencing==0.35.1,requests==2.32.3,requests-mock==1.12.1,requests-oauthlib==2.0.0,rich==13.7.1,rpds-py==0.20.0,rsa==4.9,six==1.16.0,smmap==5.0.1,snowballstemmer==2.2.0,Sphinx==7.4.7,sphinx-argparse==0.4.0,sphinx-autodoc-typehints==2.2.3,sphinx-rtd-theme==2.0.0,sphinxcontrib-applehelp==2.0.0,sphinxcontrib-devhelp==2.0.0,sphinxcontrib-htmlhelp==2.1.0,sphinxcontrib-jquery==4.1,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==2.0.0,sphinxcontrib-serializinghtml==2.0.0,stevedore==5.2.0,tomli==2.0.1,tqdm==4.66.5,types-cffi==1.16.0.20240331,types-PyMySQL==1.1.0.20240524,types-pyOpenSSL==24.1.0.20240722,types-redis==4.6.0.20240806,types-requests==2.31.0.6,types-setuptools==71.1.0.20240813,types-urllib3==1.26.25.14,typing_extensions==4.12.2,urllib3==1.26.19,websocket-client==1.8.0,# Editable Git install with no remote (wikimedia-spicerack==8.10.1.dev1+g3c39887),-e /src,wmflib==1.2.5,wrapt==1.16.0,zipp==3.20.0
00:00:57.644 py39-flake8 run-test-pre: PYTHONHASHSEED='525301326'
00:00:57.644 py39-flake8 run-test: commands[0] | flake8 setup.py spicerack
00:00:58.812 py39-unit create: /src/.tox/py39-unit

It is not necessarily a problem since eventually all those deps have to be build and once they are build they are reused. Looks like something is not cached properly cause surely wheels should have been created and cached or at least be reused between environments which is not the case, the py39-bandit env also takes 45 seconds to install deps?

00:02:17.196 py39-bandit create: /src/.tox/py39-bandit
00:02:20.846 py39-bandit installdeps: .[tests]
00:02:55.648 py39-bandit develop-inst: /src
00:03:04.584 py39-bandit installed: alabaster==0.7.16,attrs==24.2.0,babel==2.16.0,bandit==1.7.9,cachetools==5.4.0,certifi==2024.7.4,cffi==1.17.0,charset-normalizer==3.3.2,ClusterShell==1.9.2,conftool==3.2.1,coverage==7.6.1,cryptography==43.0.0,cumin==4.2.0,Deprecated==1.2.14,dnspython==2.0.0,docutils==0.20.1,elasticsearch==7.14.2,exceptiongroup==1.2.2,execnet==2.1.1,flake8==7.1.1,gitdb==4.0.11,GitPython==3.1.43,google-auth==2.33.0,idna==3.7,imagesize==1.4.1,importlib_metadata==8.2.0,iniconfig==2.0.0,Jinja2==3.1.4,jsonschema==4.23.0,jsonschema-specifications==2023.12.1,kafka-python==2.0.2,kubernetes==12.0.1,markdown-it-py==3.0.0,MarkupSafe==2.1.5,mccabe==0.7.0,mdurl==0.1.2,mypy==1.11.1,mypy-extensions==1.0.0,oauthlib==3.2.2,packaging==24.1,pbr==6.0.0,phabricator==0.9.1,pluggy==1.5.0,pyasn1==0.6.0,pyasn1_modules==0.4.0,pycodestyle==2.12.1,pycparser==2.22,pyflakes==3.2.0,Pygments==2.18.0,PyMySQL==1.1.1,pynetbox==6.6.2,pyparsing==3.1.2,pytest==8.3.2,pytest-cov==5.0.0,pytest-xdist==3.6.1,python-dateutil==2.9.0.post0,python-etcd==0.4.5,PyYAML==6.0.2,redis==4.1.3,referencing==0.35.1,requests==2.32.3,requests-mock==1.12.1,requests-oauthlib==2.0.0,rich==13.7.1,rpds-py==0.20.0,rsa==4.9,six==1.16.0,smmap==5.0.1,snowballstemmer==2.2.0,Sphinx==7.4.7,sphinx-argparse==0.4.0,sphinx-autodoc-typehints==2.2.3,sphinx-rtd-theme==2.0.0,sphinxcontrib-applehelp==2.0.0,sphinxcontrib-devhelp==2.0.0,sphinxcontrib-htmlhelp==2.1.0,sphinxcontrib-jquery==4.1,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==2.0.0,sphinxcontrib-serializinghtml==2.0.0,stevedore==5.2.0,tomli==2.0.1,tqdm==4.66.5,types-cffi==1.16.0.20240331,types-PyMySQL==1.1.0.20240524,types-pyOpenSSL==24.1.0.20240722,types-redis==4.6.0.20240806,types-requests==2.31.0.6,types-setuptools==71.1.0.20240813,types-urllib3==1.26.25.14,typing_extensions==4.12.2,urllib3==1.26.19,websocket-client==1.8.0,# Editable Git install with no remote (wikimedia-spicerack==8.10.1.dev1+g3c39887),-e /src,wmflib==1.2.5,wrapt==1.16.0,zipp==3.20.0

Then when bandit runs in 4 minutes which is a lot:

00:03:04.585 py39-bandit run-test: commands[0] | bandit -l -i -r --skip B404,B603 --exclude ./spicerack/tests ./spicerack/
...
00:04:13.803 Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:03
00:04:45.871 Run started:2024-08-14 12:31:50.400569
00:04:54.857 
00:04:54.857 Test results:
00:04:54.858 	No issues identified.
00:06:08.258 
00:06:08.258 Code scanned:
00:06:08.258 	Total lines of code: 11486
00:06:08.258 	Total lines skipped (#nosec): 4
00:06:08.259 	Total potential issues skipped due to specifically being disabled (e.g., #nosec BXXX): 0
00:06:08.259 
00:06:08.259 Run metrics:
00:06:08.259 	Total issues (by severity):
00:06:08.259 		Undefined: 0
00:06:08.259 		Low: 0
00:06:08.259 		Medium: 0
00:06:08.259 		High: 0
00:06:08.259 	Total issues (by confidence):
00:06:14.657 		Undefined: 0
00:06:18.256 		Low: 0
00:06:51.849 		Medium: 0
00:07:00.739 		High: 0
00:07:00.740 Files skipped (2):
00:07:00.740 	spicerack/tests/fixtures/cookbook/cookbooks/group3/invalid_subgroup/__init__.py (syntax error while parsing AST from file)
00:07:02.044 	spicerack/tests/fixtures/cookbook/cookbooks/group3/invalid_syntax.py (syntax error while parsing AST from file)
00:07:02.286 py39-mypy create: /src/.tox/py39-mypy

Which is bandit taking so much time? I have no idea, but surely we should run it only once.

For the CI caching issue, I'll have to investigate.

@hashar thanks a lot for the investigation! Would it be possible to have an option in the jjb config to allow Python jobs to specify the list of tox environments to execute? If so we could set the py39 ones for spicerack as special use case, and update them when the cumin nodes are upgraded.

• elukey moved this task from In Progress to Waiting for others on the User-Elukey board.Aug 27 2024, 3:04 PM

On my local machine, with warmed cache, setting up the test environment takes 43 seconds, so that is pretty similar to what CI is experiencing.

There are 21 environment defined (tox -l|wc -l), so that would be 15 minutes just to prepare the environments.
bandit takes 4 minutes, it is run under python 3.9, 3.10 and 3.11 which is 12 minutes.
That is already 27 minutes, I am not counting the time for Sphinx, prospector, unit tests, mypy etc.

Running time tox3 --notest (which does not run any command) took 13,5 minutes:

real	13m34,833s
user	10m14,645s
sys	0m50,439s

tox version 3 supports sharing an environment directory between multiple environments. That it from having to do a full pip install for each envs. https://tox.wiki/en/3.27.1/config.html#conf-envdir , that is not supported by tox v4 but given how badly broken it is, I am not recommending to upgrade to it.

style, Sphinx, MyPY, Prospector can probably run a single time, I don't think they need to run under each python versions.

A quick fix would be to change the env list when running on Jenkins, which in tox can be done via a tox:jenkins section (that is loaded whenever tox detects JENKINS_URL environment variable is set):

tox.ini

[tox:jenkins]
# If CI misses a python version we require, it should fail!
skip_missing_interpreters = false
envlist = py{39,310,311}-{flake8,unit},py39-{style,bandit,mypy,prospector,sphinx}

docker-registry.wikimedia.org/releng/tox-v3 defaults to 3.7.

A longer one is to use the envdir to share the installed environment between similar envs, and/or have some envs install less dependencies such as flake8 which really only requires flake8 and does not even need the software to be installed (which it does due to usedevelop = true).

We should pair on that next week :]

Change #1069220 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/software/spicerack@master] tox: run less environments on CI

https://gerrit.wikimedia.org/r/1069220

gerritbot added a project: Patch-For-Review.Aug 30 2024, 2:21 PM

Change #1069224 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/spicerack@master] tox: add config for jenkins

https://gerrit.wikimedia.org/r/1069224

Change #1069226 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/software/spicerack@master] tox: only install flake8 when running flake8

https://gerrit.wikimedia.org/r/1069226

Self note: in Quibble I did some optimization to sdist / install which can be seen at: https://gerrit.wikimedia.org/g/integration/quibble/+/40329a702b4e2ff5ee135d4e341a98b1e88fa3f5/tox.ini

tox.ini

[testenv]
skipsdist = True
usedevelop = True

[testenv:check]
# So we at least try sdist once
skipsdist = False
usedevelop = False
commands = python setup.py --version
           python setup.py check

And I guess black, flake8 would need to have skip_install = True cause there is no need to install the app to lint it :)

Change #1069224 abandoned by Elukey:

[operations/software/spicerack@master] tox: add config for jenkins

Reason:

superseded by https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1069226

https://gerrit.wikimedia.org/r/1069224

Change #1069220 merged by jenkins-bot:

[operations/software/spicerack@master] tox: run less environments on CI

https://gerrit.wikimedia.org/r/1069220

• elukey moved this task from Waiting for others to In Progress on the User-Elukey board.Sep 4 2024, 9:43 AM

• elukey moved this task from In Progress to Stalled on the User-Elukey board.Sep 6 2024, 1:42 PM

• elukey lowered the priority of this task from High to Medium.Sep 16 2024, 2:18 PM

Change #1069226 merged by jenkins-bot:

[operations/software/spicerack@master] tox: only install flake8 when running flake8

https://gerrit.wikimedia.org/r/1069226

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2024, 2:31 PM

• elukey closed this task as Resolved.Oct 10 2024, 7:13 AM

• elukey claimed this task.

Spicerack's tox config times out all the time after T342019Closed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Spicerack's tox config times out all the time after T342019
Closed, ResolvedPublic
Actions