In T152351#2845753, @hashar wrote:Regarding the long build time (11 minutes), that is due to the job compiling a lot of material. The cause is we only cache the tarballs downloaded from npmjs.org, I guess we should also cache node_modules and expire old modules by running npm prune. I think we tried at some point and that had side effects due to npm/node version changing. I guess we can try again. Some history from two years ago on T76304#795614
Worth forking to its own task :]
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Krinkle | T159591 Castor: mediawiki-core-qunit-jessie node_modules cache ineffective | |||
| Resolved | Krinkle | T152386 CI tests on wikimedia/portals repo: cache node_modules to save time |
Event Timeline
This comment was removed by Jdrewniak.
Comment Actions
As @hashar mentioned in his comment:
Regarding the long build time (11 minutes), that is due to the job compiling a lot of material.
So for the portal repo, we can basically try to compile less material.
The long compile times are mostly caused by one build-step gulp sprite.
This task uses the sprity npm module to merge all the images into a CSS sprite.
Sprity depends on lwip - a node image processor.
Compiling lwip is what causes the very long npm install step in CI (as well as locally).
This compilation depends on node-gyp which depends on Python 2.7 (obsolete).
... (lwip starts compiling now) 01:25:09 > lwip@0.0.9 install /home/jenkins/workspace/wikimedia-portals-npm-node-4-jessie/node_modules/sprity/node_modules/sprity-lwip/node_modules/lwip 01:25:09 > node-gyp rebuild 01:25:09 01:25:11 make: Entering directory '/home/jenkins/workspace/wikimedia-portals-npm-node-4-jessie/node_modules/sprity/node_modules/sprity-lwip/node_modules/lwip/build' ... 01:27:46 COPY Release/lwip_image.node 01:27:46 make: Leaving directory '/home/jenkins/workspace/wikimedia-portals-npm-node-4-jessie/node_modules/sprity/node_modules/sprity-lwip/node_modules/lwip/build' 01:28:19 gulp-rename@1.2.2 node_modules/gulp-rename .... (lwip finishes three minutes later )
If we want to get rid of these dependancies and shorten the CI test time, we can change the image optimize strategy. For a while, it's been suggested that we use SVGs instead of the current CSS sprites. T129634
Switching over to SVGs would remove the Sprity npm module, and thus the long compile times. We would instead use gulp-iconify which doesn't have as much compiled dependancies as Sprity.
Comment Actions
That came around on T159591 which is more generic. In short instead of doing:
rm -fR node_modules npm install
We would:
// restore node_modules npm prune // clear out obsolete versions npm update // resync!
And save node_modules when a change is merged.
Comment Actions
Moving this to stalled and to the backlog board, as we can pick this work up at a future date.
See this old check-in:
https://gerrit.wikimedia.org/r/#/c/346152/
Comment Actions
That will be done at the same time as T159591 the draft patch being https://gerrit.wikimedia.org/r/#/c/395610/ Not sure when I will be able to properly test it and deploy it. Would take a few weeks though since I am currently busy migrating CI to Docker containers.
So essentially the task for portals is a duplicate, but it does not hurt to have a more specific sub tasks to make sure the issue is solved for portals. :]