Page MenuHomePhabricator

Jobs sometimes fail with "/usr/local/bin/npm: No such file or directory"
Closed, ResolvedPublic

Description

1krinkle@integration-saltmaster:~$ sudo salt -v '*slave-trusty*' cmd.run 'which npm; npm --version; ls -l /usr/local/bin/npm'
2Executing job with jid 20160311021818841073
3-------------------------------------------
4
5integration-slave-trusty-1016.integration.eqiad.wmflabs:
6 /usr/bin/npm
7 2.14.12
8 ls: cannot access /usr/local/bin/npm: No such file or directory
9integration-slave-trusty-1005.integration.eqiad.wmflabs:
10 /usr/local/bin/npm
11 2.14.12
12 lrwxrwxrwx 1 root root 38 Mar 11 01:59 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
13integration-slave-trusty-1023.integration.eqiad.wmflabs:
14 /usr/bin/npm
15 2.14.12
16 ls: cannot access /usr/local/bin/npm: No such file or directory
17integration-slave-trusty-1015.integration.eqiad.wmflabs:
18 /usr/local/bin/npm
19 2.14.12
20 lrwxrwxrwx 1 root root 38 Mar 11 02:09 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
21integration-slave-trusty-1003.integration.eqiad.wmflabs:
22 /usr/local/bin/npm
23 2.14.12
24 lrwxrwxrwx 1 root root 38 Mar 11 02:10 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
25integration-slave-trusty-1018.integration.eqiad.wmflabs:
26 /usr/local/bin/npm
27 2.14.12
28 lrwxrwxrwx 1 root root 38 Mar 11 02:03 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
29integration-slave-trusty-1013.integration.eqiad.wmflabs:
30 /usr/bin/npm
31 2.14.12
32 ls: cannot access /usr/local/bin/npm: No such file or directory
33integration-slave-trusty-1004.integration.eqiad.wmflabs:
34 /usr/local/bin/npm
35 2.14.12
36 lrwxrwxrwx 1 root root 38 Mar 11 01:52 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
37integration-slave-trusty-1017.integration.eqiad.wmflabs:
38 /usr/bin/npm
39 2.14.12
40 ls: cannot access /usr/local/bin/npm: No such file or directory
41integration-slave-trusty-1006.integration.eqiad.wmflabs:
42 /usr/local/bin/npm
43 2.14.12
44 lrwxrwxrwx 1 root root 38 Mar 11 01:57 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
45integration-slave-trusty-1012.integration.eqiad.wmflabs:
46 /usr/bin/npm
47 2.14.12
48 ls: cannot access /usr/local/bin/npm: No such file or directory
49integration-slave-trusty-1014.integration.eqiad.wmflabs:
50 /usr/bin/npm
51 2.14.12
52 ls: cannot access /usr/local/bin/npm: No such file or directory
53integration-slave-trusty-1001.integration.eqiad.wmflabs:
54 /usr/local/bin/npm
55 2.14.12
56 lrwxrwxrwx 1 root root 38 Mar 11 02:17 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
57integration-slave-trusty-1002.integration.eqiad.wmflabs:
58 /usr/local/bin/npm
59 2.14.12
60 lrwxrwxrwx 1 root root 38 Mar 11 02:12 /usr/local/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
61integration-slave-trusty-1011.integration.eqiad.wmflabs:
62 /usr/bin/npm
63 2.14.12
64 ls: cannot access /usr/local/bin/npm: No such file or directory

Looks like the way npm was installed varies on a few instances. They all have it and the same version, but in different places (/usr/bin/npm vs /usr/local/bin/npm).

In theory that shouldn't be a problem if things use plain npm from bash, but I guess that isn't the case given some jobs randomly fail half-way stating /usr/local/bin/npm couldn't be found.

https://integration.wikimedia.org/ci/job/npm/58031/console

02:02:22 + node --version
02:02:22 v0.10.25
02:02:22 + npm --version
02:02:23 2.14.12
02:02:23 + rm -rf node_modules
02:02:23 + npm install
02:02:58 grunt-cli@0.1.13 node_modules/grunt-cli
02:02:58 ├── nopt@1.0.10 (abbrev@1.0.7)
02:02:58 ├── resolve@0.3.1
02:02:58 └── findup-sync@0.1.3 (glob@3.2.11, lodash@2.4.2)
02:02:58 
..
02:02:58 + npm test
02:02:58 /tmp/hudson5960120158220888376.sh: line 7: /usr/local/bin/npm: No such file or directory
02:02:58 Build step 'Execute shell' marked build as failure

Related Objects

StatusSubtypeAssignedTask
DuplicateNone
ResolvedKrinkle
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Declinedhashar
Resolvedhashar
Declinedhashar
Resolvedhashar
Declinedhashar
Resolvedhashar
Resolvedhashar
Resolvedmobrovac
Resolvedhashar
Resolvedhashar
Resolvedmobrovac
Resolvedhashar
DuplicateNone
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
ResolvedPaladox
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
InvalidNone
Resolvedhashar
DeclinedNone
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar

Event Timeline

Here are all the npm job builds that failed due to the no such file or directory error per node:

53102/build.xml:  <builtOn>integration-slave-trusty-1002</builtOn>
53272/build.xml:  <builtOn>integration-slave-trusty-1001</builtOn>
53491/build.xml:  <builtOn>integration-slave-trusty-1001</builtOn>
54146/build.xml:  <builtOn>integration-slave-trusty-1003</builtOn>
54536/build.xml:  <builtOn>integration-slave-trusty-1003</builtOn>
55656/build.xml:  <builtOn>integration-slave-trusty-1018</builtOn>
56130/build.xml:  <builtOn>integration-slave-trusty-1018</builtOn>
56412/build.xml:  <builtOn>integration-slave-trusty-1001</builtOn>
57214/build.xml:  <builtOn>integration-slave-trusty-1003</builtOn>
58031/build.xml:  <builtOn>integration-slave-trusty-1018</builtOn>

No idea what is going on. Maybe puppet run delete it somehow. Anyway the npm jobs are going to be migrated to Nodepool soonish and that will no more be an issue.

hashar lowered the priority of this task from Unbreak Now! to High.Mar 30 2016, 3:55 PM

Lowering priority. Having an outdated npm version is currently not a concern for the repositories that have been migrated to Nodepool since the tests/scripts invoked are fairly simple (just grunt + a few grunt tasks).

Since the .deb package is unlikely to happen anytime, I am willing to install npm from npm using the old npm to bootstrap it. Hopefully that will get it under /usr/local/bin/npm and we will have no conflict.

Krinkle closed this task as Resolved.EditedApr 1 2016, 6:48 PM
Krinkle claimed this task.

I've removed /usr/local/bin/npm from those slaves with salt so that it's now consistent across all slaves.

The crazy thing is that npm does correctly self-install to /usr/local/bin. After applying https://gerrit.wikimedia.org/r/#/c/280956/, all slaves had the updated version in /usr/local/bin/npm. However, naturally the copy at /usr/local/bin/npm was clobbering the PATH and taking precedence.

@hashar The install path is configurable in npmrc (see npm conf list, npm conf ls -l, npm conf get prefix). Setting it to /usr/local would make perfect sense, indeed. It defaults to prefix /usr for system installs, such as from Ubuntu and Debian.