Page MenuHomePhabricator

Investigate integration-jjb-config-diff job slowness
Closed, ResolvedPublic

Description

The integration-jjb-config jobs takes almost 4 minutes to run since it runs JJB twice on our rather large set of jobs.

We use tox to setup the env, it might be compiling PyYaml on each invocation which would be an easy hanging fruit.

When I ever I get extensions tested together, we will have less PHPUnit/qunit jobs for extensions.

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar changed Security from none to None.
hashar added a subscriber: hashar.

It takes the same duration when I run it locally without any wrappers (3-4 minutes). In Jenkins it runs it twice (once with and once without the proposed change) to produce a diff.

It could be, but I don't think the tox overhead is significant. It's mostly the speed of JJB itself and our use of it, namely that we have almost 3500 (three thousand five hundred!) jobs. T91997 and T91996 will help bring that down to a more manageable number - and also speed up our JJB build a lot.

hashar claimed this task.

Thanks to the reduction in the number of jobs (I think), the build time went down to less than 2 minutes!

https://integration.wikimedia.org/ci/job/integration-jjb-config-diff/buildTimeTrend

jjb-config-diff.png (400×500 px, 15 KB)

2 minutes is more acceptable, hence I am closing this task.