Page MenuHomePhabricator

Package npm 2.14
Closed, ResolvedPublic

Description

We have the latest stable Node.JS package in our repos, but not the accompanying npm package:

$ apt-cache madison npm
       npm | 1.4.21+ds-2 | http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages
       npm | 1.4.21+ds-2 | http://mirrors.wikimedia.org/debian/ jessie/main Sources
$ apt-cache madison nodejs
    nodejs | 4.2.4~dfsg-1~bpo8+1 | http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
    nodejs | 0.10.29~dfsg-2 | http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages
    nodejs | 0.10.29~dfsg-2 | http://mirrors.wikimedia.org/debian/ jessie/main Sources
    nodejs | 4.2.4~dfsg-1~bpo8+1 | http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main Sources

This causes problems when you want to do npm install (e.g.) on a Jessie box.

Unfortunately, for some reason it seems that the Debian folks haven't released a new npm package yet. A quick search reveals that stable, testing and unstable all have the same package version - 1.4.21+ds-2

Event Timeline

mobrovac created this task.Jan 22 2016, 9:25 PM
mobrovac raised the priority of this task from to Needs Triage.
mobrovac updated the task description. (Show Details)
mobrovac added projects: Operations, Services.
mobrovac added a subscriber: mobrovac.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 22 2016, 9:25 PM

Hmm, the package has been touched since 2014. There's a wishlist bug to upgrade it (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=804633), but that hasn't seen a followup either...

Node 4.x ships with its own npm. One shouldn't install "npm" from Debian as its own package. Maybe we're shadowing it somehow, due to the separate npm package overwriting the bin link?

The nodejs package from Debian does not ship npm (see file list at https://packages.debian.org/sid/amd64/nodejs/filelist ). Instead, npm is split to its own npm package. To get NodeJs 4.2/4.3 we backported the nodejs package from sid, npm hasn't been updated either by us or upstream Debian.

So this task is about extracting npm from Nodejs 4.2/4.3, update the npm package and ship it.

Meanwhile, the Jessie instances will be stuck to npm 1.4.21. I am not going to have puppet to ship it via npm install -g. That has proven to be a nightmare to handle.

Paladox set Security to None.Feb 13 2016, 10:09 PM
Paladox added a subscriber: Paladox.
mobrovac changed the task status from Open to Stalled.Feb 14 2016, 8:18 PM

I'm setting this task to stalled, as this is only relevant for edge cases where we execute npm install directly on the hosts (some testing hosts and CI). I don't think there is much value in packaging it ourselves until the version is bumped by upstream.

NOTE: Since the nodejs package is backported and the versions differ quite considerably, this may cause problems with requirements for the npm and nodejs-legacy packages. The solution is to install nodejs=0.10 along with npm and nodejs-legacy first and then apt-get install nodejs separately.
MoritzMuehlenhoff triaged this task as Low priority.Mar 1 2016, 8:14 AM

Will npm-node-4.3 CI jobs be fixed too? They actually run npm install...

Will npm-node-4.3 CI jobs be fixed too? They actually run npm install...

That job runs on I instances that are using Jessie configured to get packages from apt.wikimedia.org. So yes, whenever a new version is available the will catch up.

This comment was removed by Paladox.
Paladox added a comment.EditedMar 17 2016, 8:59 PM

@hashar helped me to figure out an easyer way of installing npm

Please do the following

Add this to package.json

"dependencies": { "npm": "~3" }

you can pick your own npm version to run.

:)

See instructions at https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#Recent_npm_version

I'm setting this task to stalled, as this is only relevant for edge cases where we execute npm install directly on the hosts (some testing hosts and CI). I don't think there is much value in packaging it ourselves until the version is bumped by upstream.

NOTE: Since the nodejs package is backported and the versions differ quite considerably, this may cause problems with requirements for the npm and nodejs-legacy packages. The solution is to install nodejs=0.10 along with npm and nodejs-legacy first and then apt-get install nodejs separately.

We can use the version directly from npm. Plus npm is running 2.x on node 0.10 but is running npm 1.x on node 4.3 so it is a down grade. We have tried npm 3.x by adding npm 3 to dependencies and it worked so no problems.

Krinkle added a comment.EditedMar 18 2016, 8:36 PM

npm is running 2.x on node 0.10 but is running npm 1.x on node 4.3 so it is a down grade. We have tried npm 3.x by adding npm 3 to dependencies and it worked so no problems.

This statement is confusing. You refer to the current situation as is the case for our Jenkins jobs named "npm" (not the software called "npm"). Our test builds that run on persistent slave indeed have Ubuntu Trusty with Node.js 0.10 and npm 2.x (Trusty defaults to 1.x but we upgraded to 2.x due to various bugs in older npm. This is fine since npm 2 supports officially supports Node 0.10). Test builds that run on our new experimental Nodepool instances have Debian Jessie with Node.js 4.x and npm 1.x.

However note that this has nothing to do with official Node.js, Debian or Ubuntu. This is all due to our own actions. Here are the upstream versions:

Official Node.js:

  • Node 0.10 ships with npm 1.x.
  • Node 0.12 ships with npm 2.x
  • Node 4.x ships with npm 2.x
  • Node 6.x (in the future) will likely ship with npm 3.x.

Package manager:

The Node v4.3.0 we currently run in Nodepool was backported by Wikimedia. But due to the lack of a packaged npm, we now have a terribly wrong combination of Node and npm.

Please do not move forward with Nodepool until npm is upgraded to at least npm 2.14.12 (the version that ships with Node v4.3.0), or npm 3.5.2 (the equivalent 3.x branch release).

Either by packaging npm, or by installing it with puppet like we have been doing for the past year on the persistent slaves https://github.com/wikimedia/operations-puppet/blob/4b78fffdfb/modules/contint/manifests/packages/javascript.pp#L39-L43

See also:

npm is running 2.x on node 0.10 but is running npm 1.x on node 4.3 so it is a down grade. We have tried npm 3.x by adding npm 3 to dependencies and it worked so no problems.

This statement is confusing. You refer to the current situation as is the case for our Jenkins jobs named "npm" (not the software called "npm"). Our test builds that run on persistent slave indeed have Ubuntu Trusty with Node.js 0.10 and npm 2.x (Trusty defaults to 1.x but we upgraded to 2.x due to various bugs in older npm. This is fine since npm 2 supports officially supports Node 0.10). Test builds that run on our new experimental Nodepool instances have Debian Jessie with Node.js 4.x and npm 1.x.

However note that this has nothing to do with official Node.js, Debian or Ubuntu. This is all due to our own actions. Here are the upstream versions:

Official Node.js:

  • Node 0.10 ships with npm 1.x.
  • Node 0.12 ships with npm 2.x
  • Node 4.x ships with npm 2.x
  • Node 6.x (in the future) will likely ship with npm 3.x.

Package manager:

The Node v4.3.0 we currently run in Nodepool was backported by Wikimedia. But due to the lack of a packaged npm, we now have a terribly wrong combination of Node and npm.

Please do not move forward with Nodepool until npm is upgraded to at least npm 2.14.12 (the version that ships with Node v4.3.0), or npm 3.5.2 (the equivalent 3.x branch release).

Either by packaging npm, or by installing it with puppet like we have been doing for the past year on the persistent slaves https://github.com/wikimedia/operations-puppet/blob/4b78fffdfb/modules/contint/manifests/packages/javascript.pp#L39-L43

See also:

Hi sorry that it was confusing. I didn't explain it as good as you did so thanks. Yes what you suggested but hashar was going to migrate all extensions but that did not happen not sure though but it might be this week. We could package nodejs 4.3 with npm 2.x that comes with it or use npm 3.

@Krinkle would a package meant for Ubuntu Xenial work on Jessie since that is debian.

@Krinkle thanks for your detailed explanation of node/npm version matching on T124474#2135319 . Yesterday I triggered the npm-node-4.3 on MediaWiki extensions and all of them passed just fine with the node 4.3 / npm 1.4.21. Most probably because for those repositories we just install very simple packages (grunt, banana, jshint etc).

Almost all repositories have been migrated now, there are still a few leftover but that is because they are invoking either jsduck or composer which are not yet provided in the Nodepool image.

npm 2.x would be nice to have, but it is currently not preventing npm jobs to run. If we end up having a blocker, we can revisit and bump this task priority. Meanwhile, it is de facto no more a blocker of T119143, migration has been almost completed and jobs are running fine.

Krinkle added a comment.EditedMar 22 2016, 8:46 PM

@hashar I'm sorry, but I can't work with that.

It passing doesn't mean it worked. Running an old npm version may mean certain hooks/tests never run in the first place, giving the false illusion of a pass.

An outdated npm also means script hooks may work differently. It may cause a build script to run twice instead of once (making the build significantly slower). It may also cause new hooks to be ignored as they didn't exist yet. For example, the js documentation generator (not migrated to Nodepool yet) would not work. It will appear to pass, but the output will be incomplete for a browser.

npm 1.4.21 is EOL, unsupported, and insecure (in general, but more so in combination with Node 4). This means we have no upstream support when we encounter bugs and other issues.

Over the past 2 years, we encountered countless bugs and performance issues with npm. Many of which I worked hard with upstream to prioritise getting fixed and subsequently upgrading our CI slaves. By downgrading us to a two year old unsupported npm release you undo all that work. It's a miracle none of the tests failed so far, but I assure you: This is a recipe for disaster.

It's nice that the services team already packaged Node, but that's no excuse to stop there. We should finish the job and package npm too. Or install it through puppet like I did for the persistent slaves, or write a shell script as part of the image build process that installs npm.

Until this is fixed, I strongly recommend all npm jobs be reverted back to the persistent slaves.

I have looked at Debian source package from https://anonscm.debian.org/git/collab-maint/nodejs.git branch master-lts and it pass the config flag --without-npm .. Ubuntu has a Debian package for npm which is fairly recent, maybe it can be published on our apt.wikimedia.org ?

Per @Krinkle earlier comment: Node 4.x ships with npm 2.x . Maybe we can get the Nodepool image to just npm install -g npm@~2 and that would be sufficient. I am pretty sure it is going to have bad interactions with the npm deb package though :(

I agree with @Krinkle here. All tests pass now is due to (1) most npm jobs running just linters for the time being; and (2) no complex npm scripts in extensions. But that's no excuse. For the CI use case, how about using nvm?

@mobrovac That might work. Nvm is most known for managing multiple node versions, but it's also a useful tool for easily installing standard Node tarballs (which include npm). It'll need to happen as part of the image creation process as running it within the job would be much too slow.

We'll need to make sure to tell nvm to set the version as default since we can't use nvms' subshell feature (Jenkins gives each execution step a fresh bash environments).

It would additionally be interesting to explore actual multi-version node (nvm's primary feature) E.g. next time there is a Node LTS release we can add it alongside the current version in Nodepool base images and create experimental Jenkins jobs to select the newer version.

This comment was removed by Paladox.

We can always use http://packages.ubuntu.com/xenial/npm which has a deb package. And since Ubuntu is based on debian it should work.

@hashar and @Krinkle what do you think of doing ^^

Or we can use https://github.com/nodejs/node/tree/master/deps/npm and add it in the deps/npm folder.

Change 280974 had a related patch set uploaded (by Hashar):
contint: stop installing grunt-cli

https://gerrit.wikimedia.org/r/280974

Change 280980 had a related patch set uploaded (by Hashar):
dib: provision npm 2.x

https://gerrit.wikimedia.org/r/280980

Change 280974 merged by Rush:
contint: stop installing grunt-cli

https://gerrit.wikimedia.org/r/280974

Mentioned in SAL [2016-04-01T20:56:58Z] <hashar> Refreshing Nodepool snapshot to hopefully get npm 2.x installed T124474

Change 280980 merged by jenkins-bot:
dib: provision npm 2.x

https://gerrit.wikimedia.org/r/280980

hashar closed this task as Resolved.Apr 1 2016, 9:43 PM
hashar claimed this task.

Nodepool images now comes with npm 2.5.12. provisioned by puppet and installed from npmjs.org.

$ /usr/bin/npm --version
1.4.21

$ /usr/local/bin/npm --version
2.15.2

Since that npm 2.x is merely for CI needs and it no more needs a debian package. I am marking this task as fixed.

Mentioned in SAL [2016-04-04T10:04:02Z] <hashar> integration: salt -v '*-slave*' cmd.run 'npm -g uninstall grunt-cli' | T124474

Mentioned in SAL [2016-04-04T10:06:01Z] <hashar> integration: salt -v '*-slave*' cmd.run 'rm /usr/local/bin/grunt; rm -fR /usr/local/lib/node_modules/grunt-cli' | T124474

Change 281427 had a related patch set uploaded (by Hashar):
Fix qunit jobs pointing to non existent 'grunt'

https://gerrit.wikimedia.org/r/281427

Change 281427 merged by jenkins-bot:
Fix qunit jobs pointing to non existent 'grunt'

https://gerrit.wikimedia.org/r/281427

It would additionally be interesting to explore actual multi-version node (nvm's primary feature) E.g. next time there is a Node LTS release we can add it alongside the current version in Nodepool base images and create experimental Jenkins jobs to select the newer version.

Indeed. Even better, we can start testing some ECMAScript 2015 features only available in Node.js 5, right now.