Is there a reason this isn't in infrastructure?

In T119143#2024057, @Jdforrester-WMF wrote:

Is there a reason this isn't in infrastructure?

No reasons really. Sometime tasks are in between infra and config. But in this case it is really infra and specially Nodepool instances.

In T119143#1843027, @hashar wrote:

A big stopper is that some repositories use Karma which expects a local setup of MediaWiki.

Not exactly. None of the repos using the npm job expect a MediaWiki install. Repos like OOjs and OOjs UI even use Grunt and Karma in their npm job and that works just fine with no outside dependencies.

The only jobs expecting MediaWiki that are somewhat npm-related are the qunit jobs, which internally use npm to fetch some of their dependencies, but it's not an "npm" job. All npm jobs should be safe to migrate to Jessie. The only outside requirements are Node and npm, and - for those running unit tests - local Chromium + Firefox browsers with Xvfb.

So the lame status update is:

Feb 1 - Feb 5
That was the week I had to rid the train. I did some work toward it on Tuesday 2nd T119143#1988025. Then staging area on deployment servers disappeared and I started what would be a 14 hours work day.
Rest of week was deployment of .wmf12

Feb 8 - Feb 12
Got rid of grunt-cli on Monday. Have hit a couple walls on Monday:

• T126246: Provision openjdk-7-jre-headless on Nodepool slaves
  • not a blocker but nice to have
• T126230: Provision pkg-config on Nodepool instances
  • Prevents node-gyp from compiling native modules. Which I found out after having migrating a couple repo to node4.2 (Graphoid and Citoid?)

Eventually we switched to php5.5 and I had to skip riding the train to focus on baby sitting CI / communicate to devs and join other CI people that nicely handled it.

Next week plan is:

• fix up the Nodepool image so it can install the packages we need
• make sure migrated repos pass fine
• migrate rest of services oid to CI

If all goes well (and there will be surprises), the generic npm job triggered for all repos will be switched.

Would have to look at whether we still want to run jobs for nodejs 0.8 and 0.10 (Parsoid comes to mind). Not task filled for that.

Let me know if you need more clarifications.

The image having npm 1.x should be a blocker for adoption. It should be npm 2.14.7 at the very least. Otherwise various scripts will work in unexpected ways, and not all tests will run properly (may result in false positive or silent failure).

grunt-cli is not needed indeed. It's deprecated as global. It's a nice clean break for migration to Jessie to make sure it's no longer relied upon.

Mentioned in SAL [2016-02-15T10:57:49Z] <hashar> bumped Nodepool disk image to fix apt configuration / include jre and package-config. Refreshing snapshot image. Poke T126246 T126230 T119143

Change 270711 had a related patch set uploaded (by Hashar):
NodeJs on Jessie got bumped 4.2 -> 4.3

https://gerrit.wikimedia.org/r/270711

Change 270711 merged by jenkins-bot:
NodeJs on Jessie got bumped 4.2 -> 4.3

https://gerrit.wikimedia.org/r/270711

Change 269142 merged by jenkins-bot:
jjb: port {name}-{repository}-npm to nodejs 4.2

https://gerrit.wikimedia.org/r/269142

Change 270870 had a related patch set uploaded (by Hashar):
Any *npm-node-4.3 jobs now put slaves offline

https://gerrit.wikimedia.org/r/270870

Change 270870 merged by jenkins-bot:
Any *npm-node-4.3 jobs now put slaves offline

https://gerrit.wikimedia.org/r/270870

Change 273135 had a related patch set uploaded (by Paladox):
Migrate OOJS repos to npm-node-4.3

https://gerrit.wikimedia.org/r/273135

Also for mediawiki this is blocked because the 4.3 version could only be run once at a time, it isent done like the npm 0.10 where it can run serval at the same time.

Change 273457 had a related patch set uploaded (by Hashar):
Migrate mediawiki npm job to Nodepool

https://gerrit.wikimedia.org/r/273457

Change 273457 merged by jenkins-bot:
Migrate mediawiki npm job to Nodepool

https://gerrit.wikimedia.org/r/273457

Mentioned in SAL [2016-02-26T15:03:09Z] <hashar> Switched MediaWiki core npm test to Nodepool instance T119143

Change 275828 had a related patch set uploaded (by Hashar):
Migrate some npm jobs to Nodepool/NodeJs 4.3

https://gerrit.wikimedia.org/r/275828

Change 275828 merged by jenkins-bot:
Migrate some npm jobs to Nodepool/NodeJs 4.3

https://gerrit.wikimedia.org/r/275828

Change 277762 had a related patch set uploaded (by Hashar):
Migrate some repos to npm-node-4.3

https://gerrit.wikimedia.org/r/277762

Change 277762 merged by jenkins-bot:
Migrate some repos to npm-node-4.3

https://gerrit.wikimedia.org/r/277762

Change 277764 had a related patch set uploaded (by Hashar):
Migrate mediawiki/skins/* to npm-node-4.3

https://gerrit.wikimedia.org/r/277764

Change 277764 merged by jenkins-bot:
Migrate mediawiki/skins/* to npm-node-4.3

https://gerrit.wikimedia.org/r/277764

Change 278003 had a related patch set uploaded (by Hashar):
zuul: introduce 'npm-legacy' template

https://gerrit.wikimedia.org/r/278003

Change 278004 had a related patch set uploaded (by Hashar):
Mass switch repositories npm job to Nodepool

https://gerrit.wikimedia.org/r/278004

Change 278005 had a related patch set uploaded (by Hashar):
Zuul: rm 'npm-node-4.3' project template

https://gerrit.wikimedia.org/r/278005

Change 278003 merged by jenkins-bot:
zuul: introduce 'npm-legacy' template

https://gerrit.wikimedia.org/r/278003

Mentioned in SAL [2016-03-21T20:28:52Z] <hashar> Mass running npm-node-4.3 jobs against MediaWiki extensions to make sure they all pass ( https://gerrit.wikimedia.org/r/#/c/278004/ | T119143 )

I have trigger the npm-node4.3 job on 311 extensions impacted by https://gerrit.wikimedia.org/r/#/c/278004/ the only failures:

Change 278004 merged by jenkins-bot:
Mass switch repositories npm job to Nodepool

https://gerrit.wikimedia.org/r/278004

Most Mediawiki extensions have been migrated for which the npm-node-4.3 yields a SUCCESS. There are still a few repo left over tagged in zuul with the project template npm-legacy.

Mentioned in SAL [2016-03-21T21:55:17Z] <hashar> zuul: almost all MediaWiki extensions migrated to run the npm job on Nodepool (with Node.js 4.3) T119143 . All tested. Will monitor the build results that ran overnight tomorrow

Change 278847 had a related patch set uploaded (by Hashar):
Move more mw extensions to Nodepool npm job

https://gerrit.wikimedia.org/r/278847

Change 278847 merged by jenkins-bot:
Move more mw extensions to Nodepool npm job

https://gerrit.wikimedia.org/r/278847

Status update

Most repositories have been migrated. What is left:

There are certainly some other left over.

Change 278005 merged by jenkins-bot:
Zuul: rm 'npm-node-4.3' project template

https://gerrit.wikimedia.org/r/278005

Change 290486 had a related patch set uploaded (by Hashar):
Migrate npm legacy test to trusty nodepool

https://gerrit.wikimedia.org/r/290486

Change 290689 had a related patch set uploaded (by Hashar):
dib: add g for NodeJS native modules compilation

https://gerrit.wikimedia.org/r/290689

Mentioned in SAL [2016-05-25T14:12:39Z] <hashar> Regenerating Nodepool snapshot to include g++ which is required by some NodeJS native modules T119143

Change 290689 merged by jenkins-bot:
dib: add g for NodeJS native modules compilation

https://gerrit.wikimedia.org/r/290689

Change 290698 had a related patch set uploaded (by Hashar):
dib: ensure_package('g ') does not work

https://gerrit.wikimedia.org/r/290698

Change 290698 merged by jenkins-bot:
dib: ensure_package('g ') does not work

https://gerrit.wikimedia.org/r/290698

Mentioned in SAL [2016-05-25T15:07:15Z] <hashar> g++ added to Jessie and Trusty Nodepool instances | T119143

Change 290486 merged by jenkins-bot:
Migrate npm legacy test to trusty nodepool

https://gerrit.wikimedia.org/r/290486

Status as of 2016-05-25

Bulk of jobs have been migrated. Have to audit what is left to migrate.

Change 290787 had a related patch set uploaded (by Hashar):
Drop old mwext-VisualEditor-npm job

https://gerrit.wikimedia.org/r/290787

Change 290787 merged by jenkins-bot:
Drop old mwext-VisualEditor-npm job

https://gerrit.wikimedia.org/r/290787

Change 290794 had a related patch set uploaded (by Hashar):
Make mwext-qunit to timeout earlier and not fail

https://gerrit.wikimedia.org/r/290794

Change 290791 had a related patch set uploaded (by Paladox):
Migrate mediawiki core jsduck test to jessie

https://gerrit.wikimedia.org/r/290791

What is left invoking npm and not on Nodepool instances? From the XML Jenkins jobs:

$ grep -l npm *|xargs grep assigned|grep -v ci-|cut -d\:  -f1
beta-parsoid-update-eqiad
castor-save
mediawiki-core-jsduck-publish
mediawiki-core-qunit
mediawiki-extensions-qunit
mwext-VisualEditor-publish
mwext-qunit
mwext-qunit-composer
oojs-core-coverage
oojs-core-publish
oojs-ui-coverage
oojs-ui-demos-publish
oojs-ui-jsduck-publish
parsoidsvc-deploy-jsduck-publish
parsoidsvc-source-jsduck-publish
performance-webpagetest-wmf
performance-webpagetest-wpt-org
unicodejs-coverage
unicodejs-publish
visualeditor-coverage

The publish / coverage jobs are out of scope. will be dealt with later on due to some architecture work that is needed to grab the material and publish it somewhere.

qunit jobs requires a local vhost / proper Mw install which is going to be worked on and hardly related to npm/javascript migration itself.

performance-* are unrelated.

So IMHO there is nothing left to migrate and the bulk of npm based jobs are now running on Nodepool. Remaining long tail will be dealt with later on.

Change 290794 abandoned by Hashar:
Make mwext-qunit to timeout earlier and not fail

Reason:
Yup definitely thank you Timo for the review :)

https://gerrit.wikimedia.org/r/290794

Change 290791 abandoned by Paladox:
Migrate mediawiki core jsduck test to jessie

https://gerrit.wikimedia.org/r/290791