Abstract
The Jenkins jobs triggered for oojs/ui.git takes too much resources on the CI infrastructure, specially when lot of changes are made in a serie and merged.
Each patchset and CR+2 trigger 3 jobs running npm, each consuming an instance. A serie of 30 patches got merged between 19:00UTC to midnight and from 17:50UTC to midnight that is 241 jobs and thus 241 instances consumed.
That is too much for the CI infrastructure to handle.
Jobs detail
When patches are proposed for the oojs/ui.git repository, we trigger three different jobs that run:
- npm test : grunt test
- npm demos : grunt publish-build demos
- npm doc : grunt build && jsduck && copy:jsduck
All of them share the CI setup overhead to clone the repository. They then run npm install which probably takes half of the build time.
Each also invoke composer install, though it benefits from caching.
Looking at the grunt tasks being run, each job also share common tasks some being rather slow (ex: colorizeSvg, svg2png:dist).
Tasks breakdown
Using the job console output:
vimdiff <(grep 'Running' npm-test.log) <(grep 'Running' npm-run-demos.log) <(grep 'Running' npm-run-doc.log)
:TOhtml
https://people.wikimedia.org/~hashar/T160513/tasksdiff.html
What would be nice
I would like all three build steps (test, doc, demos) to be unified in a single Jenkins job. That would clear out the overhead of git clone / npm install. We probably want a new task defining the tasks to run, paying attention to avoid running twice the slowest one (svg2png and colorizeSvg).
We can have Jenkins to invoke a specific npm entry point such as npm run jenkins which would have the appropriate pre/post task and maybe invoke a specific/custom grunt task.