Tool "autodesc" - can't start webservice
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Magnus
	Feb 8 2019, 3:09 PM

Description

I am trying to switch all my tools (69!) to Debian Stretch. As part of this, I stopped the webservice for the autodesc tool. However, I can't restart it, neither on trusty no scratch. Please help ASAP, tool is down.

Event Timeline

Magnus created this task.Feb 8 2019, 3:09 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 8 2019, 3:09 PM

Reedy added a project: Toolforge.Feb 8 2019, 7:19 PM

Any error messages or log output to report?

Error: The module '/mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/build/Release/contextify.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 48. This version of Node.js requires
NODE_MODULE_VERSION 57. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).
    at Object.Module._extensions..node (module.js:681:18)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at bindings (/mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/node_modules/bindings/bindings.js:81:44)
    at Object.<anonymous> (/mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/lib/contextify.js:1:96)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! autodesc@0.0.0 start: `node server.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the autodesc@0.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Looks like you need to rebuild the nodejs dependencies?

Looks like specifically, you'd need to remove /www/js/node_modules/ and run npm install again in /www/js. There's a lot installed under ~/.npm as well, but those don't appear to be part of the tool? Those may also need rebuilding. There's a big npm and nodejs version jump between trusty and stretch, not to mention any C libraries compiled against.

If the tool will be running on the Kubernetes backend, I assume you’d want to reinstall the npm dependencies from inside a webservice --backend=kubernetes nodejs shell and not on the bastion directly.

(I assume autodesc isn’t the only tool to use Node, so this would be a nice thing to add to the “solutions to common problems” section if it turns out the npm reinstall fixes the problem.)

I have run npm update in the kubernetes shell, but no joy.

If I read cat busted-20170209T1623/run_server.* correctly, it tries to run /data/project/autodesc/server.js when the file is /data/project/autodesc/www/js/server.js. Maybe that's the issue?

In T215620#4940340, @Magnus wrote:

I have run npm update in the kubernetes shell, but no joy.

$HOME/error.log ends with:

Error: The module '/mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/build/Release/contextify.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 48. This version of Node.js requires
NODE_MODULE_VERSION 57. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).
    at Object.Module._extensions..node (module.js:681:18)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at bindings (/mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/node_modules/bindings/bindings.js:112:48)
    at Object.<anonymous> (/mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/jsdom/node_modules/contextify/lib/contextify.js:1:96)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! autodesc@0.0.0 start: `node server.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the autodesc@0.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /data/project/autodesc/.npm/_logs/2019-02-09T22_07_12_817Z-debug.log

The run log referenced contains:

0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'start' ]
2 info using npm@5.8.0
3 info using node@v8.11.1
4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info lifecycle autodesc@0.0.0~prestart: autodesc@0.0.0
6 info lifecycle autodesc@0.0.0~start: autodesc@0.0.0
7 verbose lifecycle autodesc@0.0.0~start: unsafe-perm in lifecycle true
8 verbose lifecycle autodesc@0.0.0~start: PATH: /usr/share/npm/node_modules/npm-lifecycle/node-gyp-bin:/mnt/nfs/labstore-secondary-tools-project/autodesc/www/js/node_modules/.bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
9 verbose lifecycle autodesc@0.0.0~start: CWD: /mnt/nfs/labstore-secondary-tools-project/autodesc/www/js
10 silly lifecycle autodesc@0.0.0~start: Args: [ '-c', 'node server.js' ]
11 silly lifecycle autodesc@0.0.0~start: Returned: code: 1  signal: null
12 info lifecycle autodesc@0.0.0~start: Failed to exec start script
13 verbose stack Error: autodesc@0.0.0 start: `node server.js`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/usr/share/npm/node_modules/npm-lifecycle/index.js:283:16)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at EventEmitter.emit (events.js:214:7)
13 verbose stack     at ChildProcess.<anonymous> (/usr/share/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at ChildProcess.emit (events.js:214:7)
13 verbose stack     at maybeClose (internal/child_process.js:925:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
14 verbose pkgid autodesc@0.0.0
15 verbose cwd /mnt/nfs/labstore-secondary-tools-project/autodesc/www/js
16 verbose Linux 4.9.0-8-amd64
17 verbose argv "/usr/bin/node" "/usr/bin/npm" "start"
18 verbose node v8.11.1
19 verbose npm  v5.8.0
20 error code ELIFECYCLE
21 error errno 1
22 error autodesc@0.0.0 start: `node server.js`
22 error Exit status 1
23 error Failed at the autodesc@0.0.0 start script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

The 3 info using node@v8.11.1 line here tells me that this attempt to run the webservice was done on the Debian Stretch grid engine. I know this because nodejs --version on the Stretch grid is v8.11.1 while in the nodejs Kubernetes container it is v6.11.0.

The comment made by @LucasWerkmeister in T215620#4939973 may have unintentionally caused confusion. The same nodejs runtime needs to be used for both the npm commands and the webservice itself. If --backend=gridengine (the current default for webservice) is used when starting the webservice, then the associated npm operations need to be made directly on the Debian Stretch bastion. If alternatively --backend=kubernetes is intended to be used when starting the webservice the npm operations need to be performed from inside a webservice --backend=kubernetes nodejs shell runtime.

Framawiki subscribed.Feb 10 2019, 12:49 PM

Thanks, I have rebuild and updated via npm on the kubernetes shell.

New issue (I feel this one happened before somehow):

tools.autodesc@tools-sgebastion-06:~$     webservice --backend=gridengine stop
Your webservice is not running

tools.autodesc@tools-sgebastion-06:~$ webservice --backend=kubernetes nodejs start
Looks like you already have another webservice running, with a gridengine backend
You should stop that webservice by issuing:
    webservice --backend=gridengine stop
And then start it again with backend kubernetes by issuing:
    webservice --backend=kubernetes start

So the gridengine says no webservice is running, but kubernetes says it is, and won't start its own.

You might have to stop the gridengine webservice from the old bastion (login-trusty.tools.wmflabs.org).

Tried that, also on login.tools.wmflabs.org (just to be sure). Both say "webservice is not running". Still won't start kubernetes.

Simon_Villeneuve subscribed.Feb 11 2019, 2:31 PM

• SandraF_WMF subscribed.Feb 11 2019, 5:14 PM

Sjoerddebruin awarded a token.Feb 11 2019, 7:54 PM

Sjoerddebruin subscribed.

Multichill added a project: Wikidata.Feb 11 2019, 7:56 PM

In T215620#4942790, @Magnus wrote:

So the gridengine says no webservice is running, but kubernetes says it is, and won't start its own.

I added a workaround for this to the FAQ on wikitech today after it bit another tool during migration:

webservice stop (just to be sure)
rm $HOME/service.manifest
webservice [other args go here] start

That did the trick, thanks!

Tool "autodesc" - can't start webserviceClosed, ResolvedPublicActions

Description

Event Timeline

Tool "autodesc" - can't start webservice
Closed, ResolvedPublic
Actions