Maps have a specific case where we can't migrate to node 10, see T213077: Migrate Kartotherian to node-mapnik >v4.2.1 to support libmapnik >3.1.x . Because of that, maps can't use the new package created on T203239: Create Debian packages for Node.js 10 upgrade.
Since there is no estimative for mapnik to release the needed version soon, it's desirable to migrate the maps services to node 8 before node 6 LTS ends.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T321959 Tech Wishes - Maps service infrastructure deprecations | |||
| Open | None | T321789 Upgrade maps servers to node >= 14 | |||
| Duplicate | None | T210815 Upgrade kartotherian to latest upstream node-mapnik | |||
| Open | None | T213077 Migrate Kartotherian to node-mapnik >v4.2.1 to support libmapnik >3.1.x | |||
| Resolved | MSantos | T215852 Migrate Kartotherian/Tilerator to Node 10 | |||
| Invalid | None | T216521 Create Debian packages for Node.js 8 upgrade for Maps |
We can't easily maintain nodejs 8 packages in parallel, that adds a substantive maintenance overhead. Plus, 8 will be EOLed by end of the year already. What is the ETA for mapnik getting fixed? Is this an external blocker or an internal lack of resources?
@MoritzMuehlenhoff It's an external blocker. We're blocked on the release of mapnik 3.1.0, which doesn't appear to be happening anytime soon:
https://github.com/mapnik/mapnik/issues/3747
https://github.com/mapnik/mapnik/milestone/26
@MSantos As a workaround, could we run a fork of node-mapnik with the commits adding node 10 support (some or all of the commits in this PR, I think: https://github.com/mapnik/node-mapnik/pull/878) cherry-picked onto node-mapnik@v3.7.2?
This is worth a shot, IMHO. The changes in the PR are trivial enough. There are more commits in master currently, though, that target the build path (C++ and boost version compatibility code, etc) that would probably need to be ported as well.
Another alternative would be to switch to using the node-mapnik project's precompiled mapnik binaries rather than compiling from source, and using some form of sandboxing (firejail?) to mitigate the security risk (I assume security is the reason we're not using the precompiled binaries now; it's not documented anywhere). That seems to be the only mapnik installation method the node-mapnik devs are realistically supporting at this point, given that they've left mapnik 3.0.x behind, and mapnik 3.1.0 won't be released in the foreseeable future.
As things stand, we're going to run into this same problem every time we want to upgrade node versions, and the changes needed for the next node LTS might not be so easy to cherry-pick.
This branch will successfully build mapnik from source on node 10: https://github.com/mdholloway/node-mapnik/tree/v3.7.2/wikimedia
Still, it would still be much more maintainable in the long run if we could find an acceptable way to run the precompiled binaries, IMO.
Cool, I will check this out. Thanks for taking a look into that.
I am afraid that this would still be stuck on third party libraries such as mapnik-reference as long v3.1.x is not released.
A few more notes, for posterity:
After digging some more, I don't think security was the reason we've been packaging our own libmapnik debs and building mapnik from source; it's seems like it was more because the precompiled binaries are black boxes which can have unforeseen compatibility issues with our environment (see discussion at https://phabricator.wikimedia.org/T152131#2852399, for example).
As indicated by the discussion there, we were apparently using the mapnik binary blobs in production for a while, from the launch of the maps service through end of 2016. FWIW, it doesn't look like the security aspect was ever explicitly evaluated (for reference, see the security review ticket at https://phabricator.wikimedia.org/T105051).
@MoritzMuehlenhoff and @Mholloway
I tested Michael's patch and it is working great with node v10.14.2. The next step would be having the Beta Cluster set up with node 10 so I can move with further tests T215523: Upgrade node to version 10 on Beta Cluster for maps but need some help to make it happen. How can I proceed?
We'll need a puppet patch to the node10 component to the role, then we can cherrypick that patch to beta. Do we have a task for the node10 upgrade for maps?