Page MenuHomePhabricator

git-update and provisioning takes a really long time to complete
Closed, ResolvedPublic

Description

Ever since switching to restbase, running git-updateand provision now takes a really long time to complete (up to half an hour on my laptop).
We should consider solutions to alleviate this as it can represent a lot of wasted dev time.

Event Timeline

ferdbold raised the priority of this task from to Needs Triage.
ferdbold updated the task description. (Show Details)
ferdbold subscribed.

The part that seems to take the longest time for me is the full rebuilds of the npm modules when services update. With citoid, parsoid, restbase and zotero all installed on my VM that can take quite a while.

@mobrovac is there something we can do to speed that step up?

One possible optimization would be to not update the npm modules when the code for the service hasn't changed:

==> Updating citoid ...
[*] Updating repo in /vagrant/srv/citoid ...
Current branch master is up to date.
[*] Updating dependencies for citoid ...

> dtrace-provider@0.6.0 install /vagrant/srv/citoid/node_modules/bunyan/node_modules/dtrace-provider
> node scripts/install.js

content-type@1.0.1 node_modules/content-type

striptags@2.1.1 node_modules/striptags

mocha-lcov-reporter@1.0.0 node_modules/mocha-lcov-reporter

ip@1.1.0 node_modules/ip

preq@0.4.8 node_modules/preq

extend@3.0.0 node_modules/extend

tough-cookie@2.2.1 node_modules/tough-cookie

iconv-lite@0.4.13 node_modules/iconv-lite

cassandra-uuid@0.0.2 node_modules/cassandra-uuid
└── long@2.4.0

bluebird@3.1.1 node_modules/bluebird

compression@1.6.0 node_modules/compression
├── bytes@2.1.0
├── on-headers@1.0.1
├── vary@1.1.0
├── compressible@2.0.7 (mime-db@1.21.0)
├── debug@2.2.0 (ms@0.7.1)
└── accepts@1.3.1 (negotiator@0.6.0, mime-types@2.1.9)

body-parser@1.14.2 node_modules/body-parser
├── bytes@2.2.0
├── depd@1.1.0
├── qs@5.2.0
├── on-finished@2.3.0 (ee-first@1.1.1)
├── raw-body@2.1.5 (unpipe@1.0.0)
├── http-errors@1.3.1 (statuses@1.2.1, inherits@2.0.1)
├── debug@2.2.0 (ms@0.7.1)
└── type-is@1.6.11 (media-typer@0.3.0, mime-types@2.1.9)

express@4.13.3 node_modules/express
├── escape-html@1.0.2
├── merge-descriptors@1.0.0
├── array-flatten@1.1.1
├── cookie@0.1.3
├── utils-merge@1.0.0
├── cookie-signature@1.0.6
├── vary@1.0.1
├── methods@1.1.2
├── fresh@0.3.0
├── etag@1.7.0
├── range-parser@1.0.3
├── content-disposition@0.5.0
├── path-to-regexp@0.1.7
├── parseurl@1.3.1
├── depd@1.0.1
├── qs@4.0.0
├── on-finished@2.3.0 (ee-first@1.1.1)
├── finalhandler@0.4.0 (unpipe@1.0.0)
├── debug@2.2.0 (ms@0.7.1)
├── proxy-addr@1.0.10 (forwarded@0.1.0, ipaddr.js@1.0.5)
├── type-is@1.6.11 (media-typer@0.3.0, mime-types@2.1.9)
├── accepts@1.2.13 (negotiator@0.5.3, mime-types@2.1.9)
├── send@0.13.0 (destroy@1.0.3, statuses@1.2.1, ms@0.7.1, mime@1.3.4, http-errors@1.3.1)
└── serve-static@1.10.2 (escape-html@1.0.3, send@0.13.1)

mocha@2.4.5 node_modules/mocha
├── escape-string-regexp@1.0.2
├── commander@2.3.0
├── diff@1.4.0
├── supports-color@1.2.0
├── growl@1.8.1
├── debug@2.2.0 (ms@0.7.1)
├── mkdirp@0.5.1 (minimist@0.0.8)
├── glob@3.2.3 (inherits@2.0.1, graceful-fs@2.0.3, minimatch@0.2.14)
└── jade@0.26.3 (commander@0.6.1, mkdirp@0.3.0)

request@2.69.0 node_modules/request
├── aws-sign2@0.6.0
├── forever-agent@0.6.1
├── tunnel-agent@0.4.2
├── oauth-sign@0.8.1
├── is-typedarray@1.0.0
├── caseless@0.11.0
├── stringstream@0.0.5
├── isstream@0.1.2
├── json-stringify-safe@5.0.1
├── node-uuid@1.4.7
├── qs@6.0.2
├── mime-types@2.1.9 (mime-db@1.21.0)
├── combined-stream@1.0.5 (delayed-stream@1.0.0)
├── form-data@1.0.0-rc3 (async@1.5.2)
├── aws4@1.2.1 (lru-cache@2.7.3)
├── hawk@3.1.3 (cryptiles@2.0.5, boom@2.10.1, sntp@1.0.9, hoek@2.16.3)
├── bl@1.0.2 (readable-stream@2.0.5)
├── http-signature@1.1.1 (assert-plus@0.2.0, jsprim@1.2.2, sshpk@1.7.3)
└── har-validator@2.0.6 (pinkie-promise@2.0.0, commander@2.9.0, chalk@1.1.1, is-my-json-valid@2.12.4)

bunyan@1.5.1 node_modules/bunyan
├── safe-json-stringify@1.0.3
├── mv@2.1.1 (ncp@2.0.0, mkdirp@0.5.1, rimraf@2.4.5)
└── dtrace-provider@0.6.0 (nan@2.2.0)

istanbul@0.4.2 node_modules/istanbul
├── async@1.5.2
├── abbrev@1.0.7
├── esprima@2.7.2
├── wordwrap@1.0.0
├── nopt@3.0.6
├── supports-color@3.1.2 (has-flag@1.0.0)
├── once@1.3.3 (wrappy@1.0.1)
├── mkdirp@0.5.1 (minimist@0.0.8)
├── which@1.2.4 (isexe@1.1.1, is-absolute@0.1.7)
├── fileset@0.2.1 (minimatch@2.0.10, glob@5.0.15)
├── resolve@1.1.7
├── escodegen@1.7.1 (estraverse@1.9.3, esutils@2.0.2, esprima@1.2.5, optionator@0.5.0, source-map@0.2.0)
└── handlebars@4.0.5 (source-map@0.4.4, optimist@0.6.1, uglify-js@2.6.1)

cheerio@0.19.0 node_modules/cheerio
├── entities@1.1.1
├── dom-serializer@0.1.0 (domelementtype@1.1.3)
├── css-select@1.0.0 (boolbase@1.0.0, css-what@1.0.0, nth-check@1.0.1, domutils@1.4.3)
├── htmlparser2@3.8.3 (domelementtype@1.3.0, entities@1.0.0, domutils@1.5.1, readable-stream@1.1.13, domhandler@2.3.0)
└── lodash@3.10.1

mocha-jshint@2.3.0 node_modules/mocha-jshint
├── uniq@1.0.1
├── shelljs@0.4.0
├── minimatch@3.0.0 (brace-expansion@1.1.2)
└── jshint@2.8.0 (strip-json-comments@1.0.4, exit@0.1.2, console-browserify@1.1.0, minimatch@2.0.10, shelljs@0.3.0, cli@0.6.6, htmlparser2@3.8.3, lodash@3.7.0)

js-yaml@3.5.2 node_modules/js-yaml
├── esprima@2.7.2
└── argparse@1.0.4 (sprintf-js@1.0.3, lodash@4.2.1)

service-runner@0.3.8 node_modules/service-runner
├── node-txstatsd@0.1.6
├── bunyan-syslog-udp@0.1.0
├── hot-shots@2.3.1
├── gelf-stream@1.1.0 (gelfling@0.3.0)
├── yargs@3.32.0 (camelcase@2.1.0, window-size@0.1.4, y18n@3.2.0, decamelize@1.1.2, cliui@3.1.0, os-locale@1.4.0, string-width@1.0.1)
└── core-js@2.0.3

swagger-router@0.3.4 node_modules/swagger-router
├── template-expression-compiler@0.1.8
├── tassembly@0.2.3
└── core-js@2.0.3

html-metadata@1.4.1 node_modules/html-metadata
├── preq@0.3.13
├── bluebird@2.8.2
└── microdata-node@0.1.2 (is-absolute-url@1.0.0, array-unique@0.1.1, domutils@1.5.1, htmlparser2@3.9.0, es6-map@0.1.3)
[*] Restarting service citoid ...
citoid stop/waiting
citoid start/running, process 18109

This was my second run in < 5 hours:

$ time vagrant git-update
... lots and lots of stuff ...
real    21m17.821s
user    0m2.327s
sys     0m0.487s

Yeah, I'm aware of these "corner cases", it's something I've been thinking about solving. One of the main reasons why we force the rebuild even when no code has been updated is because most of the services do not require fixed versions of dependencies, but caret-style ones. As a large portion of those are being actively developed by us, sometimes we need to force a full npm run in order to get the desired outcome.

Assigning to myself for the time being to have it on my radar, but if you come up with an elegant solution to this problem feel free to steal the task.

mobrovac set Security to None.

Change 280914 had a related patch set uploaded (by Mobrovac):
git-update: Update the deps only when required

https://gerrit.wikimedia.org/r/280914

Change 280914 merged by jenkins-bot:
git-update: Update the deps only when required

https://gerrit.wikimedia.org/r/280914

mobrovac moved this task from Awaiting review to Done on the MediaWiki-Vagrant board.
mobrovac edited projects, added Wikimedia-Hackathon-2016; removed Patch-For-Review.
mobrovac removed a subscriber: gerritbot.