Page MenuHomePhabricator

Migrate Kartotherian/Tilerator to Mapnik v3.1.x when released
Open, Stalled, LowestPublic

Description

Background information

With the upcoming task T210704: Migrate node-based services in production to node10. Tilerator and Kartotherian need to upgrade some libraries to be compatible with node 10.
UPDATE: We were able to run Kartotherian/Tilerator on a Node 10 setup using a custom node-mapnik version deployed under kartotherian org. In order to get back to the upstream provider, we should still wait for mapnik v3.1.x to be released, more info below.

What

Node 10 is only supported in node-mapnik >= 4.0.0.
https://github.com/mapnik/node-mapnik/blob/master/CHANGELOG.md#400

Also, node-mapnik >= 4.0.0 works with the pre-release master branch of mapnik. That means that the migration will be blocked until mapnik v3.1.x is released (which is an ongoing work since mid-2017). When it happens, the following dependencies needs to upgrade node-mapnik to the latest version:

  • mapbox/tilelive-bridge
  • kartotherian/tilelive-vector
  • kartotherian/tilelive-tmsource
  • kartotherian/abaculus
  • kartotherian/snapshot
  • kartotherian/maki

Still, what could be tested so far indicates that one third-party dependency also needs to be improved when mapnik v3.1.x will be released and that is mapnik-reference, which doesn't support the pre-release schema yet.

How

Upgrade the libraries when the time comes.

Open questions

Is that going to be a straight forward upgrade for our dependencies? Some libraries we use are outdated and upgrade them might bring incompatibilities, this will need more tests when mapnik v3.1.x is available.

Acceptance criteria

  • Backport to Debian stretch mapnik v3.1.x when released
  • Wait for mapnik-reference to be updated on upstream
  • Upgrade node-mapnik in all dependencies
  • Test and deploy

Event Timeline

MSantos created this task.Jan 7 2019, 1:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 7 2019, 1:53 PM
MSantos renamed this task from Migrate Kartotherian/Tilerator to Node 10 to [EPIC] Migrate Kartotherian/Tilerator to Node 10.Jan 7 2019, 3:45 PM
Jhernandez renamed this task from [EPIC] Migrate Kartotherian/Tilerator to Node 10 to Migrate Kartotherian/Tilerator to Node 10.Jan 7 2019, 5:16 PM
Jhernandez triaged this task as Normal priority.
Jhernandez added a project: Epic.
Jhernandez moved this task from Backlog to Epics on the Product-Infrastructure-Team-Backlog board.
MSantos changed the task status from Open to Stalled.Feb 7 2019, 4:06 PM
Yurik added subscribers: Jhernandez, Yurik.EditedFeb 11 2019, 6:36 PM

@MSantos and @Jhernandez , would it perhaps make things far simpler to migrate to use Kubernetes/Docker? Each maps service will be able to use any version of Node, or any native package, or even a different version/distribution of Linux as needed, and it will not affect any other services even if they are running on the same physical hardware. Even migration will be easier, because you will be able to use different version of Node on the same machine by different instances of Tilerator/Kartotherian.

P.S. This approach will also allow Mapnik not to be backported because it can simply be used with an Ubunutu base image - thus simplifying managing of all dependencies.

MSantos added subscribers: Gehel, Pchelolo.EditedFeb 11 2019, 7:05 PM

@Gehel and @Pchelolo any thoughts on Yuri's comment?

Maps will go on k8s eventually afaik, but it's an independent issue. Node 6 support ends in April even for security updates and that is the primary reason for the move.

Gehel added a comment.Feb 11 2019, 7:38 PM

As said by @Pchelolo, moving to k8s would be great for a number of reason and will happen at some point. But it is a different issue. Also, k8s in the context of WMF will not mean anything can be deployed.

Yurik added a comment.EditedFeb 11 2019, 7:52 PM

@Gehel is there a doc/procedure describing limits to docker deployment? E.g. would WMF have a docker repository and a well established build process that uploads to it, does it limit to the base images like "must use base Debian but not base Alpine", etc.

My understanding of Docker deployment was "to throw things over the wall" -- developers can build a docker image, throw it to dev-ops, who can deploy it to production in the matter of hours rather than months - simply because all dependencies are baked into the built image, and because dev-ops can compartmentalize different services more easily.

Gehel added a comment.Feb 11 2019, 8:11 PM

@Gehel is there a doc/procedure describing limits to docker deployment? E.g. would WMF have a docker repository and a well established build process that uploads to it, does it limit to the base images like "must use base Debian but not base Alpine", etc.

Let's move this conversation somewhere else, this is unrelated to the current ticket. Ping me on IRC tomorrow and we'll get started.

Yurik added a comment.Feb 11 2019, 8:13 PM

@Pchelolo @Gehel I agree that Node 8 upgrade is urgent and must happen. This ticket was discussing Node 10, and mentioned that it was blocked on some dependencies, so I was proposing how to help with the difficulties of dependency upgrades, and k8s might be a good solution to that. I will be happy to chat on IRC. Thanks!!!

Sturm moved this task from Kartotherian to Tracking on the Maps board.Mar 8 2019, 11:32 AM
Sturm edited projects, added Maps; removed Maps (Kartotherian).
Sturm moved this task from Tracking to Kartographer on the Maps board.Mar 8 2019, 11:35 AM
Sturm edited projects, added Maps (Kartographer); removed Maps.
Sturm moved this task from Kartographer to Kartotherian on the Maps board.
Sturm edited projects, added Maps (Kartotherian); removed Maps (Kartographer).

@MSantos : The service::node class which is used by Kartotherian/Tilerator recently gained an option $use_nodejs10, you can extend the kartotherian/tilerator Puppet classes with a new parameter for node10 which then gets passed down to the service::node class and then enable it in deployment-prep. See modules/aqs/manifests/init.pp which already does that.

MSantos renamed this task from Migrate Kartotherian/Tilerator to Node 10 to Migrate Kartotherian/Tilerator to Node 10 when Mapnik is released to v3.1.x.May 1 2019, 8:37 PM
MSantos renamed this task from Migrate Kartotherian/Tilerator to Node 10 when Mapnik is released to v3.1.x to Migrate Kartotherian/Tilerator to Mapnik to v3.1.x when released.
MSantos lowered the priority of this task from Normal to Lowest.
MSantos renamed this task from Migrate Kartotherian/Tilerator to Mapnik to v3.1.x when released to Migrate Kartotherian/Tilerator to Mapnik v3.1.x when released.May 1 2019, 8:43 PM
MSantos updated the task description. (Show Details)