Page MenuHomePhabricator

Enable node4 for mathoid tests (mathoid/deploy tests fail)
Closed, ResolvedPublic

Description

I think that jenkins is not configured to use node 4 for the mathoid tests.
cf. https://integration.wikimedia.org/ci/job/npm/47960/console triggerd by
https://gerrit.wikimedia.org/r/#/c/262737/

However, node 4 is required for the tests to pass, and not having to merge changes manually.

Event Timeline

Physikerwelt raised the priority of this task from to High.
Physikerwelt updated the task description. (Show Details)

For the permanent slaves we have:

DistroNodes
Precisev0.8.2
Trustyv0.10.25

The disposable slaves instances are running Jessie:

DistroNodes
Jessiev0.10.29

And for one of them:

nodepool@labnodepool1001:~$ ssh jenkins@10.68.22.247 apt-cache policy nodejs
nodejs:
  Installed: 0.10.29~dfsg-2
  Candidate: 4.2.4~dfsg-1~bpo8+1
  Version table:
     4.2.4~dfsg-1~bpo8+1 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
 *** 0.10.29~dfsg-2 0
        100 /var/lib/dpkg/status

Although the snapshot we boot instance is supposed to have 4.2.4:

2016-01-21 14:16:46,003 INFO nodepool.image.build.wmflabs-eqiad.ci-jessie-wikimedia: Preparing to unpack .../nodejs-legacy_4.2.4~dfsg-1~bpo8+1_all.deb ...
2016-01-21 14:16:46,013 INFO nodepool.image.build.wmflabs-eqiad.ci-jessie-wikimedia: Unpacking nodejs-legacy (4.2.4~dfsg-1~bpo8+1) over (0.10.29~dfsg-2) ...
2016-01-21 14:16:47,699 INFO nodepool.image.build.wmflabs-eqiad.ci-jessie-wikimedia: Setting up nodejs-legacy (4.2.4~dfsg-1~bpo8+1) ...

Not sure what is happening.

Regardless, the failing job is the 'npm' one which is shared by a wide range of repositories. I have planned migrate it to the Jessie instances next week which is T119143.

That's great news, @hashar, because we moving all of the Node.JS services to Jessie and 4.2 (cf. T96017: Migrate SCA cluster to SCB (Jessie and Node 4.2)). Thanks a ton!

Found the nodejs issue. We ensure nodejs-legacy is latest but that doesn't bump nodejs:

$ apt-cache depends nodejs-legacy
nodejs-legacy
  Depends: nodejs
$ dpkg -l nodejs*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version
+++-===================-=====================
ii  nodejs              0.10.29~dfsg-2
ii  nodejs-dev          0.10.29~dfsg-2
ii  nodejs-legacy       4.2.4~dfsg-1~bpo8+1   

Nodepool spawn instances based on a snapshot which is updated by the script
https://github.com/wikimedia/integration-config/blob/1c5ca2680/nodepool/scripts/setup_node.sh

The apt-get -V -q -y upgrade complains it is holding back the nodejs and nodejs-dev packages. No clue what it means really :(

Reading package lists...
Building dependency tree...
Reading state information...
The following packages have been kept back:
   nodejs (0.10.29~dfsg-2 => 4.2.4~dfsg-1~bpo8+1)
   nodejs-dev (0.10.29~dfsg-2 => 4.2.4~dfsg-1~bpo8+1)
The following packages will be upgraded:
   gyp (0.1~svn1729-3 => 0.1+20150913git1f374df9-1~bpo8+1)
   nodejs-legacy (0.10.29~dfsg-2 => 4.2.4~dfsg-1~bpo8+1)
   zuul (2.1.0-60-g1cc37f7-wmf2jessie1 => 2.1.0-60-g1cc37f7-wmf4jessie1)
3 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 2,487 kB of archives.
After this operation, 285 kB of additional disk space will be used.
Get:1 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/thirdparty zuul amd64 2.1.0-60-g1cc37f7-wmf4jessie1 [2,021 kB]
Get:2 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports gyp all 0.1+20150913git1f374df9-1~bpo8+1 [266 kB]
Get:3 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main nodejs-legacy all 4.2.4~dfsg-1~bpo8+1 [200 kB]
Fetched 2,487 kB in 0s (15.7 MB/s)
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 3.)
debconf: falling back to frontend: Readline
Preparing to unpack .../zuul_2.1.0-60-g1cc37f7-wmf4jessie1_amd64.deb ...
Unpacking zuul (2.1.0-60-g1cc37f7-wmf4jessie1) over (2.1.0-60-g1cc37f7-wmf2jessie1) ...
Preparing to unpack .../gyp_0.1+20150913git1f374df9-1~bpo8+1_all.deb ...
Unpacking gyp (0.1+20150913git1f374df9-1~bpo8+1) over (0.1~svn1729-3) ...
Preparing to unpack .../nodejs-legacy_4.2.4~dfsg-1~bpo8+1_all.deb ...
Unpacking nodejs-legacy (4.2.4~dfsg-1~bpo8+1) over (0.10.29~dfsg-2) ...
Processing triggers for systemd (215-17+deb8u2) ...
Setting up zuul (2.1.0-60-g1cc37f7-wmf4jessie1) ...
Setting up gyp (0.1+20150913git1f374df9-1~bpo8+1) ...
Setting up nodejs-legacy (4.2.4~dfsg-1~bpo8+1) ...

Change 265802 had a related patch set uploaded (by Hashar):
nodepool: dist-upgrade when preparing snapshot

https://gerrit.wikimedia.org/r/265802

So dist-upgrade did the trick:

The following packages will be upgraded:
   gyp (0.1~svn1729-3 => 0.1+20150913git1f374df9-1~bpo8+1)
   nodejs (0.10.29~dfsg-2 => 4.2.4~dfsg-1~bpo8+1)
   nodejs-dev (0.10.29~dfsg-2 => 4.2.4~dfsg-1~bpo8+1)
   nodejs-legacy (0.10.29~dfsg-2 => 4.2.4~dfsg-1~bpo8+1)
   zuul (2.1.0-60-g1cc37f7-wmf2jessie1 => 2.1.0-60-g1cc37f7-wmf4jessie1)
Setting up nodejs (4.2.4~dfsg-1~bpo8+1) ...
Setting up nodejs-dev (4.2.4~dfsg-1~bpo8+1) ...
Setting up nodejs-legacy (4.2.4~dfsg-1~bpo8+1) ...

Image ci-jessie-wikimedia-1453492820 in wmflabs-eqiad is ready

Change 265802 merged by jenkins-bot:
nodepool: dist-upgrade when preparing snapshot

https://gerrit.wikimedia.org/r/265802

ssh jenkins@10.68.23.16 nodejs --version
v4.2.4

So Nodepool instances now have v4.2.4. Will craft a Jenkins job for that.

Change 265817 had a related patch set uploaded (by Hashar):
[mathoid] experimental npm-node-4-2

https://gerrit.wikimedia.org/r/265817

Change 265817 merged by jenkins-bot:
[mathoid] experimental npm-node-4.2

https://gerrit.wikimedia.org/r/265817

Gave it a try it fails because npm has been installed with nodejs 0.10.x and before nodejs is updated to 4.2.

The relevant part is in puppet modules/contint/manifests/packages/javascript.pp which has:

# Provide 'node' alias for 'nodejs' because Debian/Ubuntu
# already has a package called 'node'
package { 'nodejs-legacy':
    ensure => present,
    # Brings up 'nodejs' from upstream which we then override
    before => Package['npm'],
}

Which is supposed to bump nodejs but doesn't as seen in a previous comment (required dist-upgrade).

Got nodejs 4.2 provisioned. The mathoid tests seems to fail though https://integration.wikimedia.org/ci/job/npm-node-4.2/4/console :(

Mathoid needs the JRE for the tests :/

All the CI slaves have openjdk-7-jre-headless which is required by the Jenkins client : https://packages.debian.org/jessie/openjdk-7-jre-headless

Not sure why the test suite ends up throwing spawn java ENOENT

I am wrong. Java is not provisioned on the Jessie slaves. Jenkins automatically install it on the host for us though but that is not really available to others.

Change 270718 had a related patch set uploaded (by Hashar):
[mathoid] switch to nodejs 4.3.x

https://gerrit.wikimedia.org/r/270718

Change 270718 merged by jenkins-bot:
[mathoid] switch to nodejs 4.3.x

https://gerrit.wikimedia.org/r/270718

I did a recheck on the six changes that are open. 2 fails due to merge conflict.

1 passed just fine https://gerrit.wikimedia.org/r/#/c/267544

3 others fail:

https://gerrit.wikimedia.org/r/#/c/270550
https://gerrit.wikimedia.org/r/#/c/270551
https://gerrit.wikimedia.org/r/#/c/270547

Each time with the error:

+ node --version
v4.3.0
+ npm --version
1.4.21
+ npm install
npm ERR! not a package /tmp/npm-710-OrbSQLyJ/github.com/physikerwelt/MathJax-node
npm ERR! Error: ENOENT: no such file or directory, open '/tmp/npm-710-OrbSQLyJ/github.com/physikerwelt/MathJax-node-unpack/package.json'
npm ERR!     at Error (native)
npm ERR! If you need help, you may report this *entire* log,
npm ERR! including the npm and node versions, at:
npm ERR!     <http://github.com/npm/npm/issues>

npm ERR! System Linux 3.16.0-4-amd64
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install"
npm ERR! cwd /home/jenkins/workspace/npm-node-4.3
npm ERR! node -v v4.3.0
npm ERR! npm -v 1.4.21
npm ERR! path /tmp/npm-710-OrbSQLyJ/github.com/physikerwelt/MathJax-node-unpack/package.json
npm ERR! syscall open 
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! 
npm ERR! Additional logging details can be found in: 
npm ERR!     /home/jenkins/workspace/npm-node-4.3/npm-debug.log
npm ERR! not ok code 0
Build step 'Execute shell' marked build as failure

Can it fail reaching github somehow?

Change 270723 had a related patch set uploaded (by Hashar):
[mathoid/deploy] swith to NodeJS 4.3

https://gerrit.wikimedia.org/r/270723

Change 270723 merged by jenkins-bot:
[mathoid/deploy] switch to NodeJS 4.3

https://gerrit.wikimedia.org/r/270723

Can it fail reaching github somehow?

I think that was the case. The error went away after modifing the link to github,
https://gerrit.wikimedia.org/r/#/c/270547/1..2/package.json
However, there is now another error w.r.t librsvg:

14:35:25 > librsvg@0.6.1 install /home/jenkins/workspace/npm-node-4.3/node_modules/mathoid-mathjax-node/node_modules/librsvg
14:35:25 > node-gyp rebuild
14:35:25 
14:35:26 (node) child_process: options.customFds option is deprecated. Use options.stdio instead.
14:35:26 Package librsvg-2.0 was not found in the pkg-config search path.
14:35:26 Perhaps you should add the directory containing `librsvg-2.0.pc'
14:35:26 to the PKG_CONFIG_PATH environment variable
14:35:26 No package 'librsvg-2.0' found
14:35:26 Package cairo-png was not found in the pkg-config search path.
14:35:26 Perhaps you should add the directory containing `cairo-png.pc'
14:35:26 to the PKG_CONFIG_PATH environment variable
14:35:26 No package 'cairo-png' found
14:35:26 Package cairo-pdf was not found in the pkg-config search path.
14:35:26 Perhaps you should add the directory containing `cairo-pdf.pc'
14:35:26 to the PKG_CONFIG_PATH environment variable
14:35:26 No package 'cairo-pdf' found
14:35:26 Package cairo-svg was not found in the pkg-config search path.
14:35:26 Perhaps you should add the directory containing `cairo-svg.pc'
14:35:26 to the PKG_CONFIG_PATH environment variable
14:35:26 No package 'cairo-svg' found
14:35:26 gyp: Call to 'pkg-config --libs-only-l librsvg-2.0 cairo-png cairo-pdf cairo-svg' returned exit status 1 while in binding.gyp.
14:35:26 gyp ERR! configure error 
14:35:26 gyp ERR! stack Error: `gyp` failed with exit code: 1
14:35:26 gyp ERR! stack     at ChildProcess.onCpExit (/usr/share/node-gyp/lib/configure.js:344:16)
14:35:26 gyp ERR! stack     at emitTwo (events.js:87:13)
14:35:26 gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
14:35:26 gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
14:35:26 gyp ERR! System Linux 3.16.0-4-amd64
14:35:26 gyp ERR! command "/usr/bin/nodejs" "/usr/bin/node-gyp" "rebuild"
14:35:26 gyp ERR! cwd /home/jenkins/workspace/npm-node-4.3/node_modules/mathoid-mathjax-node/node_modules/librsvg
14:35:26 gyp ERR! node -v v4.3.0
14:35:26 gyp ERR! node-gyp -v v0.12.2
14:35:26 gyp ERR! not ok

@hashar: is librsvg (2.40.2-1+wm1) already deployed on instance that runs the tests?

14:35:26 gyp: Call to 'pkg-config --libs-only-l librsvg-2.0 cairo-png cairo-pdf cairo-svg' returned exit status 1 while in binding.gyp.

Cant find those dev dependencies. Lets track adding Cairo dev dependencies in another task :-)

Physikerwelt claimed this task.

I'll create that bug, if https://github.com/wikimedia/MathJax-node/pull/16 is discussed and will be merged.
This issue is resolved now, cf. https://gerrit.wikimedia.org/r/#/c/270745/.
Thank you @hashar!

I tried mediawiki/services/mathoid/deploy repository on https://gerrit.wikimedia.org/r/#/c/269178/

It fails on https://integration.wikimedia.org/ci/job/mathoid-deploy-npm-node-4.3/3/consoleFull with:

+ . npm-oid.env
++ NPM_SET_PATH=./src
++ NODE_PATH=:/home/jenkins/workspace/mathoid-deploy-npm-node-4.3/node_modules
++ PATH=/usr/local/bin:/usr/bin:/bin:/usr/games:/home/jenkins/workspace/mathoid-deploy-npm-node-4.3/node_modules/.bin
+ echo 'Injecting dev dependencies from source repo into deploy node_modules'
Injecting dev dependencies from source repo into deploy node_modules
+ /srv/deployment/integration/slave-scripts/bin/npm-install-dev.py
+ cd ./src 
+ npm test 

> mathoid@0.4.1 test /home/jenkins/workspace/mathoid-deploy-npm-node-4.3/src
> mocha

sh: 1: mocha: not found
npm ERR! Test failed.  See above for more details. 
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian
 
npm ERR! not ok code 0
Build step 'Execute shell' marked build as failure
Archiving artifacts
[PostBuildScript] - Execution post build scripts. 
[PostBuildScript] Build is not success : do not execute script
Finished: FAILURE

The git submodule update did run.

The build can not find mocha though the dev dependencies are supposed to be injected. The nom-install-dev.py script has been written by @mobrovac exactly to inject them... But apparently it is not running anything, we should see a message such as: [*] NPM devDependencies Installation [*].

On the instance, the file /srv/deployment/integration/slave-scripts/bin/npm-install-dev.py is empty as well as all other slave-script. That is a provisioning issue, I am recreating the image.

The CI slave scripts are now properly provisioned from https://integration.wikimedia.org/ci/job/mathoid-deploy-npm-node-4.3/5/consoleFull the /bin/npm-install-dev.py script finally inject the dev dependencies which are installed.

And that fails again:

[*] NPM devDependencies Installation [*]
[*] - Installing mocha-lcov-reporter ...
[*] - Installing extend ...
[*] - Installing istanbul ...
[*] - Installing dom-compare ...
[*] - Installing mocha ...
[*] - Installing xmldom ...
[*] - Installing swagger-router ...
[*] - Installing mocha-jshint ...
[*] - Installing commander ...
[*] Fixing up the server config to run in CI ...
[*] NPM devDependencies Done [*] 
+ cd ./src 
+ npm test 

> mathoid@0.4.1 test /home/jenkins/workspace/mathoid-deploy-npm-node-4.3/src
> mocha



  express app
server options changed; restarting
npm ERR! Test failed.  See above for more details. 
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian
 
npm ERR! not ok code 0

The CI part itself seems to be done. I have no idea why mediawiki/services/mathoid/deploy fails tests though :(

The CI part itself seems to be done. I have no idea why mediawiki/services/mathoid/deploy fails tests though :(

I'll take a look at this. Probably a config issue, but need to confirm.

hashar removed hashar as the assignee of this task.Mar 8 2016, 1:56 PM
hashar renamed this task from Enable node4 for mathoid tests to Enable node4 for mathoid tests (mathoid/deploy tests fail).Mar 10 2016, 1:12 PM
mobrovac changed the task status from Open to Stalled.Mar 14 2016, 9:59 AM

@hashar not much I can do until I get access to the instances failing the jobs :/

This is how I reproduce it on my local machine:

cd mediawiki/services/mathoid/deploy
export NPM_SET_PATH=./src
export NODE_PATH=`pwd`/node_modules
export PATH=`pwd`/node_modules/.bin:$PATH

# install dev dependencies
~/projects/integration/jenkins/bin/npm-install-dev.py

Then fire npm test in the src directory:

> mathoid@0.6.1 test /Users/amusso/projects/mediawiki/services/mathoid/deploy/src
> mocha

{ [Error: Cannot find module './build/Release/DTraceProviderBindings'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module './build/default/DTraceProviderBindings'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module './build/Debug/DTraceProviderBindings'] code: 'MODULE_NOT_FOUND' }


  express app
server options changed; restarting
npm ERR! Test failed.  See above for more details.

The DTrace stuff most probably being caused by bunyan and it is apparently "harmless".

I have dig a bit more in node_modules/service-runner/lib/base_service.js which has logic to load modules. I have added a few console log and:

  express app
server options changed; restarting

_requireModule( ./app.js )
Exception { [Error: Cannot find module './app.js'] code: 'MODULE_NOT_FOUND' }

_requireModule( /Users/amusso/projects/mediawiki/services/mathoid/deploy/app.js )
Exception { [Error: Cannot find module '/Users/amusso/projects/mediawiki/services/mathoid/deploy/app.js'] code: 'MODULE_NOT_FOUND' }

_requireModule( /Users/amusso/projects/mediawiki/services/mathoid/deploy/node_modules/app.js )
Exception { [Error: Cannot find module '/Users/amusso/projects/mediawiki/services/mathoid/deploy/node_modules/app.js'] code: 'MODULE_NOT_FOUND' }

I have noticed the forged config.yam points:

module: /Users/amusso/projects/mediawiki/services/mathoid/deploy/src/app.js

But lot of places are reference the config.dev.yaml file which has a relative path instead:

module: ./app.js

If I change that path, the suite progress toward another error.

In short

The npm-install-dev.py only adjusts the module path in config.yaml but the test suite / utils point at config-dev.yaml.

Change 277505 had a related patch set uploaded (by Hashar):
(DO NOT SUBMIT) Debug module require on CI

https://gerrit.wikimedia.org/r/277505

Change 277508 had a related patch set uploaded (by Mobrovac):
npm-install-dev.py: Use config.dev.yaml instead of config.yaml

https://gerrit.wikimedia.org/r/277508

Change 277508 merged by jenkins-bot:
npm-install-dev.py: Use config.dev.yaml instead of config.yaml

https://gerrit.wikimedia.org/r/277508

Mentioned in SAL [2016-03-15T14:08:14Z] <hashar> Deploying slave script change https://gerrit.wikimedia.org/r/#/c/277508/ "npm-install-dev.py: Use config.dev.yaml instead of config.yaml" for T124447

Mentioned in SAL [2016-03-15T14:14:32Z] <hashar> Refreshing Nodepool snapshot images so it get a fresh copy of slave-scripts T124447

Mentioned in SAL [2016-03-15T14:19:25Z] <hashar> Image ci-jessie-wikimedia-1458051246 in wmflabs-eqiad is ready T124447

I have hacked the job to show a few things:

14:30:55 + ls -l src/config.dev.yaml src/config.prod.yaml src/config.yaml
14:30:55 -rw-r--r-- 1 jenkins jenkins 2239 Mar 15 14:30 src/config.dev.yaml
14:30:55 -rw-r--r-- 1 jenkins jenkins 1063 Mar 15 14:30 src/config.prod.yaml
14:30:55 lrwxrwxrwx 1 jenkins jenkins   15 Mar 15 14:30 src/config.yaml -> config.dev.yaml

14:30:55 + grep app src/config.dev.yaml src/config.prod.yaml src/config.yaml
14:30:55 src/config.dev.yaml:    module: /home/jenkins/workspace/mathoid-deploy-npm-node-4.3/src/app.js
14:30:55 src/config.prod.yaml:    module: ./app.js
14:30:55 src/config.yaml:    module: /home/jenkins/workspace/mathoid-deploy-npm-node-4.3/src/app.js

Change 277505 abandoned by Hashar:
(DO NOT SUBMIT) Debug module require on CI

https://gerrit.wikimedia.org/r/277505

mobrovac removed a project: Patch-For-Review.
mobrovac removed a subscriber: gerritbot.

The tests are now passing, Merci beaucoup @hashar for your help!