Ever since switching to restbase, running git-updateand provision now takes a really long time to complete (up to half an hour on my laptop).
We should consider solutions to alleviate this as it can represent a lot of wasted dev time.
Description
Details
Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
git-update: Update the deps only when required | mediawiki/vagrant | master | +23 -8 |
Event Timeline
The part that seems to take the longest time for me is the full rebuilds of the npm modules when services update. With citoid, parsoid, restbase and zotero all installed on my VM that can take quite a while.
@mobrovac is there something we can do to speed that step up?
One possible optimization would be to not update the npm modules when the code for the service hasn't changed:
==> Updating citoid ... [*] Updating repo in /vagrant/srv/citoid ... Current branch master is up to date. [*] Updating dependencies for citoid ... > dtrace-provider@0.6.0 install /vagrant/srv/citoid/node_modules/bunyan/node_modules/dtrace-provider > node scripts/install.js content-type@1.0.1 node_modules/content-type striptags@2.1.1 node_modules/striptags mocha-lcov-reporter@1.0.0 node_modules/mocha-lcov-reporter ip@1.1.0 node_modules/ip preq@0.4.8 node_modules/preq extend@3.0.0 node_modules/extend tough-cookie@2.2.1 node_modules/tough-cookie iconv-lite@0.4.13 node_modules/iconv-lite cassandra-uuid@0.0.2 node_modules/cassandra-uuid └── long@2.4.0 bluebird@3.1.1 node_modules/bluebird compression@1.6.0 node_modules/compression ├── bytes@2.1.0 ├── on-headers@1.0.1 ├── vary@1.1.0 ├── compressible@2.0.7 (mime-db@1.21.0) ├── debug@2.2.0 (ms@0.7.1) └── accepts@1.3.1 (negotiator@0.6.0, mime-types@2.1.9) body-parser@1.14.2 node_modules/body-parser ├── bytes@2.2.0 ├── depd@1.1.0 ├── qs@5.2.0 ├── on-finished@2.3.0 (ee-first@1.1.1) ├── raw-body@2.1.5 (unpipe@1.0.0) ├── http-errors@1.3.1 (statuses@1.2.1, inherits@2.0.1) ├── debug@2.2.0 (ms@0.7.1) └── type-is@1.6.11 (media-typer@0.3.0, mime-types@2.1.9) express@4.13.3 node_modules/express ├── escape-html@1.0.2 ├── merge-descriptors@1.0.0 ├── array-flatten@1.1.1 ├── cookie@0.1.3 ├── utils-merge@1.0.0 ├── cookie-signature@1.0.6 ├── vary@1.0.1 ├── methods@1.1.2 ├── fresh@0.3.0 ├── etag@1.7.0 ├── range-parser@1.0.3 ├── content-disposition@0.5.0 ├── path-to-regexp@0.1.7 ├── parseurl@1.3.1 ├── depd@1.0.1 ├── qs@4.0.0 ├── on-finished@2.3.0 (ee-first@1.1.1) ├── finalhandler@0.4.0 (unpipe@1.0.0) ├── debug@2.2.0 (ms@0.7.1) ├── proxy-addr@1.0.10 (forwarded@0.1.0, ipaddr.js@1.0.5) ├── type-is@1.6.11 (media-typer@0.3.0, mime-types@2.1.9) ├── accepts@1.2.13 (negotiator@0.5.3, mime-types@2.1.9) ├── send@0.13.0 (destroy@1.0.3, statuses@1.2.1, ms@0.7.1, mime@1.3.4, http-errors@1.3.1) └── serve-static@1.10.2 (escape-html@1.0.3, send@0.13.1) mocha@2.4.5 node_modules/mocha ├── escape-string-regexp@1.0.2 ├── commander@2.3.0 ├── diff@1.4.0 ├── supports-color@1.2.0 ├── growl@1.8.1 ├── debug@2.2.0 (ms@0.7.1) ├── mkdirp@0.5.1 (minimist@0.0.8) ├── glob@3.2.3 (inherits@2.0.1, graceful-fs@2.0.3, minimatch@0.2.14) └── jade@0.26.3 (commander@0.6.1, mkdirp@0.3.0) request@2.69.0 node_modules/request ├── aws-sign2@0.6.0 ├── forever-agent@0.6.1 ├── tunnel-agent@0.4.2 ├── oauth-sign@0.8.1 ├── is-typedarray@1.0.0 ├── caseless@0.11.0 ├── stringstream@0.0.5 ├── isstream@0.1.2 ├── json-stringify-safe@5.0.1 ├── node-uuid@1.4.7 ├── qs@6.0.2 ├── mime-types@2.1.9 (mime-db@1.21.0) ├── combined-stream@1.0.5 (delayed-stream@1.0.0) ├── form-data@1.0.0-rc3 (async@1.5.2) ├── aws4@1.2.1 (lru-cache@2.7.3) ├── hawk@3.1.3 (cryptiles@2.0.5, boom@2.10.1, sntp@1.0.9, hoek@2.16.3) ├── bl@1.0.2 (readable-stream@2.0.5) ├── http-signature@1.1.1 (assert-plus@0.2.0, jsprim@1.2.2, sshpk@1.7.3) └── har-validator@2.0.6 (pinkie-promise@2.0.0, commander@2.9.0, chalk@1.1.1, is-my-json-valid@2.12.4) bunyan@1.5.1 node_modules/bunyan ├── safe-json-stringify@1.0.3 ├── mv@2.1.1 (ncp@2.0.0, mkdirp@0.5.1, rimraf@2.4.5) └── dtrace-provider@0.6.0 (nan@2.2.0) istanbul@0.4.2 node_modules/istanbul ├── async@1.5.2 ├── abbrev@1.0.7 ├── esprima@2.7.2 ├── wordwrap@1.0.0 ├── nopt@3.0.6 ├── supports-color@3.1.2 (has-flag@1.0.0) ├── once@1.3.3 (wrappy@1.0.1) ├── mkdirp@0.5.1 (minimist@0.0.8) ├── which@1.2.4 (isexe@1.1.1, is-absolute@0.1.7) ├── fileset@0.2.1 (minimatch@2.0.10, glob@5.0.15) ├── resolve@1.1.7 ├── escodegen@1.7.1 (estraverse@1.9.3, esutils@2.0.2, esprima@1.2.5, optionator@0.5.0, source-map@0.2.0) └── handlebars@4.0.5 (source-map@0.4.4, optimist@0.6.1, uglify-js@2.6.1) cheerio@0.19.0 node_modules/cheerio ├── entities@1.1.1 ├── dom-serializer@0.1.0 (domelementtype@1.1.3) ├── css-select@1.0.0 (boolbase@1.0.0, css-what@1.0.0, nth-check@1.0.1, domutils@1.4.3) ├── htmlparser2@3.8.3 (domelementtype@1.3.0, entities@1.0.0, domutils@1.5.1, readable-stream@1.1.13, domhandler@2.3.0) └── lodash@3.10.1 mocha-jshint@2.3.0 node_modules/mocha-jshint ├── uniq@1.0.1 ├── shelljs@0.4.0 ├── minimatch@3.0.0 (brace-expansion@1.1.2) └── jshint@2.8.0 (strip-json-comments@1.0.4, exit@0.1.2, console-browserify@1.1.0, minimatch@2.0.10, shelljs@0.3.0, cli@0.6.6, htmlparser2@3.8.3, lodash@3.7.0) js-yaml@3.5.2 node_modules/js-yaml ├── esprima@2.7.2 └── argparse@1.0.4 (sprintf-js@1.0.3, lodash@4.2.1) service-runner@0.3.8 node_modules/service-runner ├── node-txstatsd@0.1.6 ├── bunyan-syslog-udp@0.1.0 ├── hot-shots@2.3.1 ├── gelf-stream@1.1.0 (gelfling@0.3.0) ├── yargs@3.32.0 (camelcase@2.1.0, window-size@0.1.4, y18n@3.2.0, decamelize@1.1.2, cliui@3.1.0, os-locale@1.4.0, string-width@1.0.1) └── core-js@2.0.3 swagger-router@0.3.4 node_modules/swagger-router ├── template-expression-compiler@0.1.8 ├── tassembly@0.2.3 └── core-js@2.0.3 html-metadata@1.4.1 node_modules/html-metadata ├── preq@0.3.13 ├── bluebird@2.8.2 └── microdata-node@0.1.2 (is-absolute-url@1.0.0, array-unique@0.1.1, domutils@1.5.1, htmlparser2@3.9.0, es6-map@0.1.3) [*] Restarting service citoid ... citoid stop/waiting citoid start/running, process 18109
This was my second run in < 5 hours:
$ time vagrant git-update ... lots and lots of stuff ... real 21m17.821s user 0m2.327s sys 0m0.487s
Yeah, I'm aware of these "corner cases", it's something I've been thinking about solving. One of the main reasons why we force the rebuild even when no code has been updated is because most of the services do not require fixed versions of dependencies, but caret-style ones. As a large portion of those are being actively developed by us, sometimes we need to force a full npm run in order to get the desired outcome.
Assigning to myself for the time being to have it on my radar, but if you come up with an elegant solution to this problem feel free to steal the task.
Change 280914 had a related patch set uploaded (by Mobrovac):
git-update: Update the deps only when required