Implement Node6 support for Kartotherian/Tilerator
Closed, ResolvedPublic

Yurik created this task.Nov 9 2016, 7:36 PM
Restricted Application added a project: Discovery. · View Herald TranscriptNov 9 2016, 7:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Yurik moved this task from Backlog to In progress on the Maps-Sprint board.Nov 9 2016, 7:38 PM

I just built a new Kartotherian version using @Pchelolo's new service-runner code and config, and tried running it on maps-test2004 from the user space, and had this result:

[2016-11-13T04:14:21.867Z] FATAL: kartotherian/16885 on maps-test2004: Cannot find module '/home/yurik/kartotherian/node_modules/mapnik/lib/binding/node-v46-linux-x64/mapnik_settings.js'
    Error: Cannot find module '/home/yurik/kartotherian/node_modules/mapnik/lib/binding/node-v46-linux-x64/mapnik_settings.js'
        at Function.Module._resolveFilename (module.js:325:15)
        at Function.Module._load (module.js:276:25)
        at Module.require (module.js:353:17)
        at require (internal/module.js:12:17)
        at Object.<anonymous> (/home/yurik/kartotherian/node_modules/mapnik/lib/mapnik.js:8:16)
        at Module._compile (module.js:409:26)
        at Object.Module._extensions..js (module.js:416:10)
        at Module.load (module.js:343:32)
        at Function.Module._load (module.js:300:12)
        at Module.require (module.js:353:17)
        at require (internal/module.js:12:17)
        at /home/yurik/kartotherian/src/routes/kartotherian.js:29:24
        at Object.init (/home/yurik/kartotherian/node_modules/kartotherian-core/lib/core.js:64:18)
        at /home/yurik/kartotherian/src/routes/kartotherian.js:27:14
        at tryCatcher (/home/yurik/kartotherian/node_modules/bluebird/js/release/util.js:16:23)
        at Function.Promise.attempt.Promise.try (/home/yurik/kartotherian/node_modules/bluebird/js/release/method.js:39:29)

Running it under node6.9.1 on the same sever produced this:

[2016-11-13T04:19:02.945Z] FATAL: kartotherian/17910 on maps-test2004: libmapnik.so.3.0: cannot open shared object file: No such file or directory
    Error: libmapnik.so.3.0: cannot open shared object file: No such file or directory
        at Error (native)
        at Object.Module._extensions..node (module.js:597:18)
        at Module.load (module.js:487:32)
        at tryModuleLoad (module.js:446:12)
        at Function.Module._load (module.js:438:3)
        at Module.require (module.js:497:17)
        at require (internal/module.js:20:19)
        at Object.<anonymous> (/home/yurik/kartotherian/node_modules/mapnik/lib/mapnik.js:31:15)
        at Module._compile (module.js:570:32)
        at Object.Module._extensions..js (module.js:579:10)
        at Module.load (module.js:487:32)
        at tryModuleLoad (module.js:446:12)
        at Function.Module._load (module.js:438:3)
        at Module.require (module.js:497:17)
        at require (internal/module.js:20:19)
        at /home/yurik/kartotherian/src/routes/kartotherian.js:29:24
Yurik added a comment.Nov 13 2016, 4:23 AM

To test kartotherian locally, without downloading a database, use quick start

Pchelolo added a comment.EditedNov 14 2016, 6:26 PM

The first error is because it's running node v4 while the package was built for node v6, so it's expected.

The second one though means that it's building a dynamic library and it doesn't find one of the dependencies, libmapnik.so.3.0 in this case. We need to package it and install on the server the version that @faidon built https://people.wikimedia.org/~faidon/mapnik/libmapnik3.0_3.0.12+ds-2~bpo8+1_amd64.deb

It worked before without installing the package because the precompiled binaries were selfcontained - everything it copied inside the tarball: https://github.com/mapnik/node-mapnik/blob/master/scripts/build_against_sdk.sh#L84-L94 and we build just the node-mapnik and not copy deps. I think it's better to install .deb packages for libmapnik

Yurik added a comment.Nov 17 2016, 7:24 PM

Today @Gehel and I did some basic Kartotherian tests using user-space node6 and Faidon's package, and it seemed to be working well. Next steps would be to deploy the deb package on all servers (seems to be non-conflicting with the current Kartotherian), upgrade the maps-test cluster to node6, do some more testing, and move towards the production cluster.

@Yurik That's excellent news, thank you for your help on this. Now we know we're good to go with node6, but maps won't be updated very soon since the plan is to start from RESTBase, Parsoid and SCB and only afterwards move to maps.

Yurik removed a project: Maps.Dec 15 2016, 4:37 AM
debt added a subscriber: Pnorman.Jan 4 2017, 7:12 PM
Deskana raised the priority of this task from Normal to High.Jan 23 2017, 10:30 PM
Deskana added a subscriber: Deskana.

Identified as a "must have" in T155601: Stabilizing Interactive Products.

The primary issue here causing this task to be stalled is issues of compatibility with GeoJSON.

Deskana reassigned this task from Yurik to MaxSem.Feb 8 2017, 7:06 PM

Max is working on this, or more specifically the subtask T152131: "geojson" plugin is missing in Mapnik package which blocks this.

Mentioned in SAL (#wikimedia-operations) [2017-02-17T19:20:25Z] <gehel> upgrading maps-test2004 to nodejs6 for testing - T150354

The fact that we've got it building is great! Obviously there is lots of testing next, so let's see what happens when we deploy to the test cluster and in other test instances.

Change 340480 had a related patch set uploaded (by Gehel; owner: Guillaume Lederrey):
[operations/puppet] maps - fix nodejs 6 version

https://gerrit.wikimedia.org/r/340480

Change 340480 merged by Gehel:
maps - fix nodejs 6 version

https://gerrit.wikimedia.org/r/340480

The plan is for the maps service to be updated to Node 6 some time next week.

Change 342256 had a related patch set uploaded (by Gehel):
[operations/puppet] maps - upgrade maps-test cluster to node js version 6

https://gerrit.wikimedia.org/r/342256

Mentioned in SAL (#wikimedia-operations) [2017-03-10T19:19:18Z] <gehel> upgrading kartotherian on maps-test2004 - T150354

Change 342256 merged by Gehel:
[operations/puppet] maps - upgrade maps-test cluster to node js version 6

https://gerrit.wikimedia.org/r/342256

Mentioned in SAL (#wikimedia-operations) [2017-03-22T11:04:16Z] <gehel> upgrade maps to nodejs 6 - T150354

Mentioned in SAL (#wikimedia-operations) [2017-03-22T11:05:03Z] <gehel> disabling puppet on all maps servers - T150354

Change 344107 had a related patch set uploaded (by Gehel):
[operations/puppet] maps - remove the pinning of nodejs version

https://gerrit.wikimedia.org/r/344107

Change 344107 merged by Gehel:
[operations/puppet] maps - remove the pinning of nodejs version

https://gerrit.wikimedia.org/r/344107

Mentioned in SAL (#wikimedia-operations) [2017-03-22T11:53:32Z] <gehel> maps codfw fully upgraded to nodejs 6, starting upgrade on maps eqiad - T150354

Mentioned in SAL (#wikimedia-operations) [2017-03-22T12:09:43Z] <gehel> maps upgrade to nodejs 6 completed - T150354

Mentioned in SAL (#wikimedia-operations) [2017-03-22T12:20:21Z] <gehel> maps restarting kartotherian - T150354

MaxSem closed this task as Resolved.Mar 22 2017, 12:57 PM

The migration is over. I would like to express my eternal gratitude to everyone who participated.

debt awarded a token.Mar 22 2017, 3:33 PM
Deskana moved this task from In progress to Done on the Maps-Sprint board.Apr 19 2017, 4:26 PM