Page MenuHomePhabricator

webservice for nodejs fails
Closed, ResolvedPublic

Description

I have been running a nodejs based tool for quite a while now at
https://tools.wmflabs.org/autodesc/

Today, I noticed it was unresponsive, so I did

webservice restart

Which reported OK, but all I get is "404 not found". tried gridengine and kubernetes, specified nodejs, stop/start, no joy.

Did some requirement change? New path needed?

This tool is used by several others, which all fail now, to a degree. Please help!

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Hi @Magnus I tried a few things to get this going and at the moment I'm seeing 502 bad gateway instead of 404.

tools.autodesc@tools-bastion-03:~$ webservice nodejs status
Your webservice is not running

webservice nodejs start

If I look at error.log I do see:

> > autodesc@0.0.0 start /mnt/nfs/labstore-secondary-tools-project/autodesc/www/js
> node server.js

/mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/node_modules/bindings/bindings.js:83
        throw e
              ^
Error: /mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/build/Release/contextify.node: undefined symbol: node_module_register
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at bindings (/mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/node_modules/bindings/bindings.js:76:44)
    at Object.<anonymous> (/mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/lib/contextify.js:1:96)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
npm ERR! weird error 8
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian

npm ERR! not ok code 0

tools.autodesc@tools-bastion-03:~$ nodejs -v
v0.10.25

Then I see a series of new job ID's as the webservice is auto-restarted I believe:

970166
970174
970183

I see more:

autodesc@0.0.0 start /mnt/nfs/labstore-secondary-tools-project/autodesc/www/js
node server.js

npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian
npm ERR! not ok code 0

Precise example node version:

rush@tools-webgrid-lighttpd-1201:~$ nodejs -v
v0.8.2

Tried npm update, which worked fine, but no joy.

  1. Can I update this myself? Probably not.
  1. Why did this suddenly fail? Who changed what?

Trusty node version example

rush@tools-webgrid-lighttpd-1413:~$ nodejs -v
v0.10.25

With "this", I meant "npm start" and the subsequent call to "node"

@Magnus, do you have an idea how long it's been down for?

Tried npm update, which worked fine, but no joy.

  1. Can I update this myself? Probably not.

I'm wondering if there isn't precise/trusty behavior selection here this resulting in this.

  1. Why did this suddenly fail? Who changed what?

I haven't found a change in this arena yet that would otherwise explain. I'm wondering when this started?

I think there is a possibility that things were broken by a combination of a nodejs package upgrade in the WMF apt repo (T149331: Node 6 upgrade planning), the base Docker images used for Kubernetes being rebuilt on 2017-02-08, and a Docker restart to upgrade the underlying runtime (https://tools.wmflabs.org/sal/log/AVocMWWhHQCSeVEJfJFA).

The new nodejs version is v6.9.1:

tools-bastion-02.tools:~
bd808$ sudo become autodesc
tools.autodesc@tools-bastion-02:~$ webservice --backend=kubernetes nodejs shell
If you don't see a command prompt, try pressing enter.
tools.autodesc@interactive:~$
tools.autodesc@interactive:~$ node --version
v6.9.1
tools.autodesc@tools-bastion-02:~$ webservice --backend=kubernetes nodejs shell
If you don't see a command prompt, try pressing enter.
tools.autodesc@interactive:~$
tools.autodesc@interactive:~$ ls
busted-20170209T1623  logs         README          service.manifest  www
error.log             public_html  replica.my.cnf  tmp
tools.autodesc@interactive:~$ cd www/js/
tools.autodesc@interactive:/data/project/autodesc/www/js$ ls
auto_long_desc.js  infobox_generator.js  package.json   short_autodesc.js
index.html         node_modules          reasonator.js  test.js
infoboxes.json     npm-debug.log         server.js      wikidata.js
tools.autodesc@interactive:/data/project/autodesc/www/js$ node server.js
/data/project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/node_modules/bindings/bindings.js:83
        throw e
        ^

Error: Module version mismatch. Expected 48, got 46.
    at Error (native)
    at Object.Module._extensions..node (module.js:597:18)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at bindings (/data/project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/node_modules/bindings/bindings.js:76:44)
    at Object.<anonymous> (/data/project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/lib/contextify.js:1:96)
    at Module._compile (module.js:570:32)
tools.autodesc@interactive:/data/project/autodesc/www/js$ mv node_modules/ node_modules-busted
tools.autodesc@interactive:/data/project/autodesc/www/js$ npm install
(node:39) fs: re-evaluating native module sources is not supported. If you are using the graceful-fs module, please update it to a more recent version.
npm WARN package.json autodesc@0.0.0 No repository field.
npm WARN package.json autodesc@0.0.0 No README data
\
> contextify@0.1.15 install /data/project/autodesc/www/js/node_modules/jsdom/node_modules/contextify
> node-gyp rebuild

(node:50) fs: re-evaluating native module sources is not supported. If you are using the graceful-fs module, please update it to a more recent version.
(node:50) DeprecationWarning: child_process: options.customFds option is deprecated. Use options.stdio instead.
make: Entering directory '/data/project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/build'
  CXX(target) Release/obj.target/contextify/src/contextify.o
  SOLINK_MODULE(target) Release/obj.target/contextify.node
  COPY Release/contextify.node
make: Leaving directory '/data/project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/build'
http@0.0.0 node_modules/http

qs@2.3.3 node_modules/qs

url@0.10.3 node_modules/url
├── punycode@1.3.2
└── querystring@0.2.0

jquery@2.1.4 node_modules/jquery

jsdom@3.1.2 node_modules/jsdom
├── xml-name-validator@1.0.0
├── xmlhttprequest@1.8.0
├── browser-request@0.3.3
├── nwmatcher@1.3.9
├── cssom@0.3.2
├── parse5@1.5.1
├── acorn@0.11.0
├── acorn-globals@1.0.9 (acorn@2.7.0)
├── htmlparser2@3.9.2 (domelementtype@1.3.0, inherits@2.0.3, entities@1.1.1, domhandler@2.3.0, readable-stream@2.2.2, domutils@1.5.1)
├── escodegen@1.8.1 (esprima@2.7.3, estraverse@1.9.3, esutils@2.0.2, optionator@0.8.2, source-map@0.2.0)
├── contextify@0.1.15 (bindings@1.2.1, nan@2.5.1)
├── request@2.79.0 (tunnel-agent@0.4.3, aws-sign2@0.6.0, oauth-sign@0.8.2, forever-agent@0.6.1, caseless@0.11.0, is-typedarray@1.0.0, stringstream@0.0.5, aws4@1.6.0, isstream@0.1.2, json-stringify-safe@5.0.1, extend@3.0.0, uuid@3.0.1, qs@6.3.0, combined-stream@1.0.5, tough-cookie@2.3.2, mime-types@2.1.14, form-data@2.1.2, hawk@3.1.3, http-signature@1.1.1, har-validator@2.0.6)
└── cssstyle@0.2.37
tools.autodesc@interactive:/data/project/autodesc/www/js$ node server.js
internal/net.js:17
    throw new RangeError('"port" argument must be >= 0 and < 65536');
    ^

RangeError: "port" argument must be >= 0 and < 65536
    at assertPort (internal/net.js:17:11)
    at Server.listen (net.js:1388:5)
    at Object.<anonymous> (/data/project/autodesc/www/js/server.js:82:4)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)
tools.autodesc@interactive:/data/project/autodesc/www/js$ logout
Session ended, resume using 'kubectl attach interactive -c interactive -i -t' command when the pod is running
Pod stopped. Session cannot be resumed.

tools.autodesc@tools-bastion-02:~$ webservice --backend=kubernetes nodejs start
Starting webservice...
tools.autodesc@tools-bastion-02:~$ kubectl get pods
NAME                        READY     STATUS    RESTARTS   AGE
autodesc-3932480877-ubfe5   1/1       Running   0          12s

That looks better.

So the problem was that nodejs was updated and the node modules needed to be rebuilt to work with the new version of the runtime.

bd808 claimed this task.

I sent an email to labs-l about this issue as there are likely other tools that are affected: https://lists.wikimedia.org/pipermail/labs-l/2017-February/004921.html