Page MenuHomePhabricator

Kartotherian node development blocked by mapnik failure
Closed, ResolvedPublic2 Estimated Story Points

Description

Problem

npm commands fail on most kartotherian projects. For example,

cd kartotherian/packages/snapshot
npm install

Result:

... (much compiling ensues)
../src/mapnik_vector_tile.hpp:11:37: fatal error: vector_tile_merc_tile.hpp: No such file or directory
compilation terminated.
mapnik.target.mk:240: recipe for target 'Release/obj.target/mapnik/src/node_mapnik.o' failed
make: *** [Release/obj.target/mapnik/src/node_mapnik.o] Error 1
make: Leaving directory '/srv/kartotherian/packages/snapshot/node_modules/@kartotherian/mapnik/build'

The error changes every time. I've tried a lot of things with no success.

Ideally, we can make mapnik build reliably, but many have tried and perished on this quest. As a last resort, we might upgrade mapnik and its library from upstream, see for example T210815. This might give us access to the precompiled node-gyp binaries.

Implementation

We found that the forked mapnik's node-gyp-pre settings could be manipulated such that it behaved like the magic --build-from-source npm client flag is passed everywhere. This lets us simplify away that detail from our client usages.

Then, the attached gerrit patches can be merged.

Event Timeline

awight renamed this task from Mitigate failure to build the mapnik module to Kartotherian node development blocked by mapnik failure.Feb 10 2022, 1:02 PM

This work overlaps with (task TBD) creating a builder image for kartotherian.

I'll experiment with a docker image to build mapnik alone. If that works, it can be integrated with the WMDE Technical Wishes docker-dev container and with a kartotherian builder image, as a multi-stage docker image or with the dependencies embedded in the other images.

One thread to pull on is to unravel why "lerna" is capable of installing mapnik, when a direct "npm" is not.

Change 761641 had a related patch set uploaded (by Awight; author: Awight):

[mediawiki/services/kartotherian@master] [WIP] node-mapnik is always built from source

https://gerrit.wikimedia.org/r/761641

It was as simple as hoped. By passing npm install --build-from-source=@kartotherian/mapnik, the same option as set in lerna.json, all of the "npm-pre-gyp" downloading is skipped.

My recommendations:

awight moved this task from Tech Review to Doing on the WMDE-TechWish-Sprint-2022-02-02 board.

This beast is rearing its head again, and the flags above fail to placate the compiler.

Deprioritizing again—my latest issue was that I had mixed dependencies by trying to upgrade mapnik to the upstream version.

Change 764790 had a related patch set uploaded (by Awight; author: Awight):

[mediawiki/services/kartotherian@master] Use patched forks to help mapnik build

https://gerrit.wikimedia.org/r/764790

Some of the code review can be done by our team, so I'm moving this back to our review column.

awight set the point value for this task to 2.

Testing in first-level dependent repos, the change in mapnik-3.7.3-wikimedia.0 seems to do the trick, so the remaining patches are to propagate the change through the dependency graph.

Change 761641 abandoned by Awight:

[mediawiki/services/kartotherian@master] node-mapnik is always built from source

Reason:

Won't be needed after the monolith refactor.

https://gerrit.wikimedia.org/r/761641

Change 768192 had a related patch set uploaded (by Awight; author: Awight):

[mediawiki/services/kartotherian@master] [DNM] Temporary workaround for stale mapnik dependencies

https://gerrit.wikimedia.org/r/768192

Change 764790 abandoned by Awight:

[mediawiki/services/kartotherian@master] Use patched forks to help mapnik build

Reason:

done better in I9e0e60920ba20e44840e1c158d78908fa799b189

https://gerrit.wikimedia.org/r/764790

awight claimed this task.
awight updated the task description. (Show Details)

Change 768192 abandoned by Awight:

[mediawiki/services/kartotherian@master] [DNM] Temporary workaround for stale mapnik dependencies

Reason:

https://gerrit.wikimedia.org/r/768192