The main thing that will affect us that was deprecated in npm 2 and removed in npm 3 is the automatic installation of peerDependencies. As of npm 3, these must be specified in repos directly.

Deprecation warnings like these have been shown in Jenkins and for developers locally for a long time (and presumably anyone working with standard Node 6 or Node 7 locally will have had to fix the repo as those come with npm 3).

npm WARN peerDependencies The peer dependency stylelint@7.8.0 included from stylelint-config-wikimedia will no
npm WARN peerDependencies longer be automatically installed to fulfill the peerDependency
npm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly.

But, still, not all repos have been fixed.

Digging in the Jenkins build logs:

$ grep -Z -l peerDependencies /srv/jenkins/builds/*npm*/*/log|xargs -0 -n1 grep 'Cloning repository'|cut -d\/ -f4-|sort|uniq -c
     18 labs/tools/heritage
      6 mediawiki/extensions/3D
      2 mediawiki/extensions/BetaFeatures
      2 mediawiki/extensions/BoilerPlate
     13 mediawiki/extensions/Cite
      7 mediawiki/extensions/CodeEditor
     49 mediawiki/extensions/CodeMirror
      3 mediawiki/extensions/Collection
     23 mediawiki/extensions/ConfirmEdit
     75 mediawiki/extensions/Echo
      2 mediawiki/extensions/ElectronPdfService
      3 mediawiki/extensions/ExtensionDistributor
     16 mediawiki/extensions/FlaggedRevs
     39 mediawiki/extensions/Flow
      9 mediawiki/extensions/Math
    143 mediawiki/extensions/MobileFrontend
      8 mediawiki/extensions/MultimediaViewer
     80 mediawiki/extensions/ORES
     11 mediawiki/extensions/Pickle
    191 mediawiki/extensions/Popups
     11 mediawiki/extensions/ProofreadPage
     72 mediawiki/extensions/RevisionSlider
      1 mediawiki/extensions/Score
      3 mediawiki/extensions/SyntaxHighlight_GeSHi
     16 mediawiki/extensions/TimedMediaHandler
     55 mediawiki/extensions/TwoColConflict
     22 mediawiki/extensions/UniversalLanguageSelector
      5 mediawiki/extensions/UploadWizard
    261 mediawiki/extensions/VisualEditor
     11 mediawiki/extensions/WikiEditor
      2 mediawiki/extensions/wikihiero
     29 mediawiki/extensions/Wikispeech
      1 mediawiki/services/mobileapps
      6 mediawiki/services/mobileapps/deploy
     32 mediawiki/services/trending-edits
     27 mediawiki/services/trending-edits/deploy
      1 mediawiki/skins/apex
      4 mediawiki/skins/Blueprint
      3 mediawiki/skins/MonoBook
     16 mediawiki/skins/Vector
     66 oojs/ui
      9 performance/WebPageTest
    396 VisualEditor/VisualEditor
      7 wikidata/query/gui

The large majority of them being due to stylelint not being explicitly listed as a dependency of stylelint-config-standard:

$ grep -h  'peerDependencies The peer' /srv/jenkins/builds/*npm*/*/log|sort|uniq -c
      6 npm WARN peerDependencies The peer dependency eslint@>=0.8.0 included from eslint-plugin-jsdoc will no
      1 npm WARN peerDependencies The peer dependency eslint-plugin-jsdoc@^3.0.0 included from eslint-config-node-services will no
      9 npm WARN peerDependencies The peer dependency jscs@^2.1.1 included from mocha-jscs will no
     59 npm WARN peerDependencies The peer dependency jscs@^3.0.3 included from mocha-jscs will no
     18 npm WARN peerDependencies The peer dependency stylelint@^6.4.0 included from stylelint-config-standard will no
    367 npm WARN peerDependencies The peer dependency stylelint@^7.0.2 included from stylelint-config-wikimedia will no
   1883 npm WARN peerDependencies The peer dependency stylelint@7.8.0 included from stylelint-config-wikimedia will no
      7 npm WARN peerDependencies The peer dependency webpack@1 || 2 || ^2.1.0-beta || ^2.2.0-rc included from babel-loader will no

How hard would it be to get a modern version of npm in CI?

We could do npm install -g npm@3 or npm install -g npm before the node image is built.

As exposed above, npm 3 resolves dependencies slightly differently than npm 2. Namely all the warnings about peer dependencies would cause the repo to fail.

The differences are explained on https://docs.npmjs.com/how-npm-works/npm3

So I guess we need:

• someone to draft an announce to wikitech-l
• find out repositories that trigger the alarm (can be done by grepping the jenkins log)
• define the peer dependencies explicitly for all branches we support (eg for mediawiki repos the REL* branches)

Then the upgrade of npm is straightforward:

14     # DO NOT CHANGE VERSION WITHOUT INVOLVING Krinkle OR hashar
15     #
16     # https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup
17     $versions = {
18         'npm'       => '2.15.2',
19     }
20

Gotta be bumped and merged in puppet.

Then force refresh the Jessie image in Nodepool https://wikitech.wikimedia.org/wiki/Nodepool#Manually_generate_a_new_snapshot

$ ssh labnodepool1001.eqiad.wmnet
user$ become-nodepool
nodepool$ git -C /etc/nodepool/wikimedia/ pull
...
nodepool$ nodepool image-update wmflabs-eqiad snapshot-co-jessie

Once the pool of jessie instances have been consumed, new instances will spawn with the refreshed image and use npm 3.x

Change 368459 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/puppet@production] [DNM] ContInt: Upgrade npm from 2.15.2 to 3.8.3 in CI

https://gerrit.wikimedia.org/r/368459

Draft:

CI npm version upgraded from 2.x to 3.x

Please note that we've upgraded the version of npm used in the Wikimedia continuous integration infrastructure. This means that CI should be more consistent with developers' local set-ups, and lets us test in a more normal environment. This change will only affect node-based CI tasks, like the standard eslint/stylelint/jsonlint/banana tasks.

This is a breaking change for npm, as it will no longer silently install peer dependencies – you need to specify them manually. If your repo was previously warning about peer dependencies, it may now no longer work. If you have this situation, you can fix it by adding the appropriate peer dependencies to your repo's package.json file, and if you can't work out how to fix it feel free to ask me in IRC.

Thanks all!

[Feel free to re-write into something better.]

(Should we jump to npm 4 or 5 instead?)

We should upgrade to npm 5 :).

Also +1 to your mock up email :)

@Jdforrester-WMF thank you for the mail draft!!! It looks good to me.

npm 4 has a few breaking changes but nothing that should impact us. Though we never know what might happen.

npm 5 http://blog.npmjs.org/post/161081169345/v500 comes with a new cache system. So there is a bunch of cache cleanup to handle.

I would prefer to be conservative and do the upgrade incrementally.

I have run the grep for peerDependencies on contint1001, all repos in P5820 would need peer dependencies to be explicitly defined. So I guess we can solve those and then upgrade to 3.

I have created sub task T172094 to attach Gerrit changes to it. That is to avoid spamming this task.

I think I got rid of the peer dependencies found in the repositories master branches. Other branches would probably require some cherry pick / similar cleanup or the npm job would end up falling (was T172094).

@Jdforrester-WMF I guess I will do the npm upgrade tomorrow during european morning then copy paste your announce to wikitech-l.

Sounds good.

@hashar did you upgrade to npm3 during the morning? :)

Na I did a bunch of tests this morning and caught up a few more peer dependencies issues :/

Change 368459 merged by Ayounsi:
[operations/puppet@production] contint: upgrade npm from 2.15.2 to 3.8.3 in CI

https://gerrit.wikimedia.org/r/368459

Mentioned in SAL (#wikimedia-releng) [2017-08-01T15:34:52Z] <hashar> Refreshing nodepool Jessie image to bump npm from 2.x to 3.8.x T161861

Mentioned in SAL (#wikimedia-releng) [2017-08-01T15:44:04Z] <hashar> Debug: Executing '/usr/bin/npm install -g npm@3.8.3' - T161861

Mentioned in SAL (#wikimedia-releng) [2017-08-01T15:45:51Z] <hashar> Image snapshot-ci-jessie-1501601670 in wmflabs-eqiad is ready && purging old instances T161861

npm 3.8.3 is now on the CI instances \O/

Can we mark as Resolved then?