Page MenuHomePhabricator
Create Task
Maniphest T129617

Jobs sometimes fail with "/usr/local/bin/npm: No such file or directory"
Closed, ResolvedPublic
Actions

Description

P2743 (An Untitled Masterwork)
1krinkle@integration-saltmaster:~$ sudo salt -v '*slave-trusty*' cmd.run 'which npm; npm --version; ls -l /usr/local/bin/npm'
2Executing job with jid 20160311021818841073
3-------------------------------------------
4
5integration-slave-trusty-1016.integration.eqiad.wmflabs:
6 /usr/bin/npm
7 2.14.12
8 ls: cannot access /usr/local/bin/npm: No such file or directory
9integration-slave-trusty-1005.integration.eqiad.wmflabs:
10 /usr/local/bin/npm
11 2.14.12
12 lrwxrwxrwx 1 root root 38 Mar 11 01:59 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
13integration-slave-trusty-1023.integration.eqiad.wmflabs:
14 /usr/bin/npm
15 2.14.12
16 ls: cannot access /usr/local/bin/npm: No such file or directory
17integration-slave-trusty-1015.integration.eqiad.wmflabs:
18 /usr/local/bin/npm
19 2.14.12
20 lrwxrwxrwx 1 root root 38 Mar 11 02:09 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
21integration-slave-trusty-1003.integration.eqiad.wmflabs:
22 /usr/local/bin/npm
23 2.14.12
24 lrwxrwxrwx 1 root root 38 Mar 11 02:10 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
25integration-slave-trusty-1018.integration.eqiad.wmflabs:
26 /usr/local/bin/npm
27 2.14.12
28 lrwxrwxrwx 1 root root 38 Mar 11 02:03 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
29integration-slave-trusty-1013.integration.eqiad.wmflabs:
30 /usr/bin/npm
31 2.14.12
32 ls: cannot access /usr/local/bin/npm: No such file or directory
33integration-slave-trusty-1004.integration.eqiad.wmflabs:
34 /usr/local/bin/npm
35 2.14.12
36 lrwxrwxrwx 1 root root 38 Mar 11 01:52 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
37integration-slave-trusty-1017.integration.eqiad.wmflabs:
38 /usr/bin/npm
39 2.14.12
40 ls: cannot access /usr/local/bin/npm: No such file or directory
41integration-slave-trusty-1006.integration.eqiad.wmflabs:
42 /usr/local/bin/npm
43 2.14.12
44 lrwxrwxrwx 1 root root 38 Mar 11 01:57 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
45integration-slave-trusty-1012.integration.eqiad.wmflabs:
46 /usr/bin/npm
47 2.14.12
48 ls: cannot access /usr/local/bin/npm: No such file or directory
49integration-slave-trusty-1014.integration.eqiad.wmflabs:
50 /usr/bin/npm
51 2.14.12
52 ls: cannot access /usr/local/bin/npm: No such file or directory
53integration-slave-trusty-1001.integration.eqiad.wmflabs:
54 /usr/local/bin/npm
55 2.14.12
56 lrwxrwxrwx 1 root root 38 Mar 11 02:17 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
57integration-slave-trusty-1002.integration.eqiad.wmflabs:
58 /usr/local/bin/npm
59 2.14.12
60 lrwxrwxrwx 1 root root 38 Mar 11 02:12 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
61integration-slave-trusty-1011.integration.eqiad.wmflabs:
62 /usr/bin/npm
63 2.14.12
64 ls: cannot access /usr/local/bin/npm: No such file or directory

Looks like the way npm was installed varies on a few instances. They all have it and the same version, but in different places (/usr/bin/npm vs /usr/local/bin/npm).

In theory that shouldn't be a problem if things use plain npm from bash, but I guess that isn't the case given some jobs randomly fail half-way stating /usr/local/bin/npm couldn't be found.

https://integration.wikimedia.org/ci/job/npm/58031/console

02:02:22 + node --version
02:02:22 v0.10.25
02:02:22 + npm --version
02:02:23 2.14.12
02:02:23 + rm -rf node_modules
02:02:23 + npm install
02:02:58 grunt-cli@0.1.13 node_modules/grunt-cli
02:02:58 ├── nopt@1.0.10 (abbrev@1.0.7)
02:02:58 ├── resolve@0.3.1
02:02:58 └── findup-sync@0.1.3 (glob@3.2.11, lodash@2.4.2)
02:02:58 
..
02:02:58 + npm test
02:02:58 /tmp/hudson5960120158220888376.sh: line 7: /usr/local/bin/npm: No such file or directory
02:02:58 Build step 'Execute shell' marked build as failure

Related Objects
Search...

StatusSubtypeAssignedTask
 DuplicateNoneT125017 Switch Parsoid to Jessie and Node 4.2
 ResolvedKrinkleT129617 Jobs sometimes fail with "/usr/local/bin/npm: No such file or directory"
 ResolvedhasharT119143 Migrate javascript npm CI jobs to Nodepool
 ResolvedhasharT113903 Puppetize npm/grunt manual setup
 ResolvedhasharT95003 Use systemd for xvfb service on Debian/Jessie
 ResolvedhasharT112560 [tracking] Disposable VMs need a cache for package managers
 ResolvedhasharT112561 Evaluate angry-caching-proxy as a package managers cache
 ResolvedhasharT114871 Evaluate devpi for caching Pypi python packages
 DeclinedhasharT116015 Investigate using a Squid based man in the middle proxy to cache package manager SSL connections
 ResolvedhasharT116017 Investigate using a cache store/restore system for package managers
 DeclinedhasharT116038 Investigate using Drydock for CI
 ResolvedhasharT117207 Puppetize / play out with devpi
 DeclinedhasharT119196 Investigate using gemstash
 ResolvedhasharT126991 Migrate services CI jobs to node 4.3
 ResolvedhasharT106668 Enable jenkins test & submit for graphoid repo
 Resolved• mobrovacT119693 Extract Graphoid Debian packages dependencies to a standalone puppet class so they can be installed on CI slaves
 ResolvedhasharT129092 Hiera is not properly configured on Nodepool instances
 ResolvedhasharT126230 Provision pkg-config on Nodepool instances
 Resolved• mobrovacT124447 Enable node4 for mathoid tests (mathoid/deploy tests fail)
 ResolvedhasharT126246 Provision openjdk-7-jre-headless on Nodepool slaves
 DuplicateNoneT129880 Give @mobrovac access to CI instances
 ResolvedhasharT126992 Migrate Parsoid CI jobs to Nodepool and add Node 4.3 support
 ResolvedhasharT124474 Package npm 2.14
 ResolvedhasharT126995 Migrate mediawiki/services/mobileapps CI jobs from node 0.10 to 4.3
 ResolvedhasharT127447 Migrate cxserver CI jobs to nodeJs 4.3
 ResolvedPaladoxT128089 Migrate OOjs npm CI job to Nodepool
 ResolvedhasharT128091 Migrate OOjs UI npm, npm-run-doc and npm-run-demos CI jobs to Nodepool
 ResolvedhasharT128092 Provide composer on the nodepool servers so OOjs UI can use it in the npm job
 ResolvedhasharT128093 Migrate VisualEditor core npm CI job to Nodepool
 ResolvedhasharT128090 Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing
 ResolvedhasharT129320 Diskimage fail on xvfb/systemd
 ResolvedhasharT129345 Xvfb service does not start on Nodepool instances
 InvalidNoneT128135 Jessie should have the ability to run several tests at once like UbuntuTrusty can
 ResolvedhasharT130207 Migrate pywikibot/i18n npm job to Nodepool
 DeclinedNoneT130208 Jobs should checkout source in a subdirectory of the workspace to avoid polluting it with CI related files
 ResolvedhasharT130587 Migrate MobileFrontend npm job to Nodepool
 ResolvedhasharT109005 Switch CI from jsduck deb package to a gemfile/bundler system
 ResolvedhasharT136096 Create jsduck job that runs on Nodepool instances
 ResolvedhasharT132144 Migrate mediawiki/extensions/Collection/OfflineContentGenerator npm jobs to Nodepool instances / Node 4.3

Event Timeline

Krinkle created this task.Mar 11 2016, 2:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2016, 2:20 AM
hashar subscribed.Mar 11 2016, 9:31 AM

Here are all the npm job builds that failed due to the no such file or directory error per node:

53102/build.xml:  <builtOn>integration-slave-trusty-1002</builtOn>
53272/build.xml:  <builtOn>integration-slave-trusty-1001</builtOn>
53491/build.xml:  <builtOn>integration-slave-trusty-1001</builtOn>
54146/build.xml:  <builtOn>integration-slave-trusty-1003</builtOn>
54536/build.xml:  <builtOn>integration-slave-trusty-1003</builtOn>
55656/build.xml:  <builtOn>integration-slave-trusty-1018</builtOn>
56130/build.xml:  <builtOn>integration-slave-trusty-1018</builtOn>
56412/build.xml:  <builtOn>integration-slave-trusty-1001</builtOn>
57214/build.xml:  <builtOn>integration-slave-trusty-1003</builtOn>
58031/build.xml:  <builtOn>integration-slave-trusty-1018</builtOn>

No idea what is going on. Maybe puppet run delete it somehow. Anyway the npm jobs are going to be migrated to Nodepool soonish and that will no more be an issue.

hashar added a subtask: T119143: Migrate javascript npm CI jobs to Nodepool.Mar 11 2016, 9:33 AM
greg merged a task: T130320: npm jobs sometimes fail with Cannot find module '/usr/local/bin/npm'.Mar 24 2016, 5:32 PM
greg added subscribers: Paladox, Catrope.
hashar lowered the priority of this task from Unbreak Now! to High.Mar 30 2016, 3:55 PM

Lowering priority. Having an outdated npm version is currently not a concern for the repositories that have been migrated to Nodepool since the tests/scripts invoked are fairly simple (just grunt + a few grunt tasks).

Since the .deb package is unlikely to happen anytime, I am willing to install npm from npm using the old npm to bootstrap it. Hopefully that will get it under /usr/local/bin/npm and we will have no conflict.

Krinkle closed this task as Resolved.EditedApr 1 2016, 6:48 PM
Krinkle claimed this task.

I've removed /usr/local/bin/npm from those slaves with salt so that it's now consistent across all slaves.

The crazy thing is that npm does correctly self-install to /usr/local/bin. After applying https://gerrit.wikimedia.org/r/#/c/280956/, all slaves had the updated version in /usr/local/bin/npm. However, naturally the copy at /usr/local/bin/npm was clobbering the PATH and taking precedence.

@hashar The install path is configurable in npmrc (see npm conf list, npm conf ls -l, npm conf get prefix). Setting it to /usr/local would make perfect sense, indeed. It defaults to prefix /usr for system installs, such as from Ubuntu and Debian.

hashar closed subtask T119143: Migrate javascript npm CI jobs to Nodepool as Resolved.May 25 2016, 9:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL