Page MenuHomePhabricator

JJB is too slow
Closed, ResolvedPublic


JJB takes age to expand the yaml files. It should be faster.

I went profiling a run of jjb for our config using:

$ cd integration/config
$ tox -e jenkins-jobs --notest
$ . ./.tox/jenkins-jobs/bin/activate
(jenkins-jobs)$ python3 -m cProfile -o jenkins-jobs --conf jenkins_jobs.ini test ./jjb/ -o /tmp/new

The resulting can be analyzed with python -m cprofile or more elegantly with

Most of the time is spend loading pkg_resources entry points.

Event Timeline

hashar triaged this task as Medium priority.Oct 28 2019, 10:10 PM
hashar added a project: Upstream.
hashar moved this task from Backlog to Patch available upstream on the Jenkins board.
PatchDescriptionBeforeAfter preload entrypoints before caching28 seconds17 seconds cache component loading17 seconds4 seconds


Jdforrester-WMF changed the task status from Open to Stalled.Oct 31 2019, 7:21 PM
Jdforrester-WMF added a subscriber: Jdforrester-WMF.

Stalled waiting on upstream merge.

Both changes got merged. Pending release.

hashar changed the task status from Stalled to Open.Nov 6 2019, 8:47 AM
hashar claimed this task.
hashar moved this task from Backlog to Patch merged upstream on the Upstream board.

Change 548964 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 3.1.0

Change 548964 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 3.1.0

Before 3.1.0:

$ time ./utils/
real	1m25,691s
user	1m22,462s
sys	0m1,487s


$ time ./utils/
real	0m12,512s
user	0m10,481s
sys	0m0,829s

85 seconds down to 12 seconds for two run of jjb and diffing. I guess that is enough to claim this task resolved.

utils/ can further be improved by caching result of the run against HEAD^ patch, but one thing at a time. 12 seconds is good enough.