Page MenuHomePhabricator

Migrate Kartotherian to node-mapnik >v4.2.1 to support libmapnik >3.1.x
Open, Needs TriagePublic

Description

Background information

We currently work around some version dependencies and run Kartotherian on Node 10, by using a custom node-mapnik build deployed under the kartotherian GitHub org.

A way to get rid of that fork is described and explored in T327396: Migrate Kartotherian to node-mapnik v4.2.1 and unfork. Getting back to upstream node-mapnik will allow us to further upgrade to versions more compatible to libmapnik v3.1.x

What & How

The following dependencies need to upgrade node-mapnik to the latest version. All of these are compatible with node-mapnik v4.2.1. That's the version of our current fork:

  • kartotherian/node-mapnik => Unforked see T327396: Migrate Kartotherian to node-mapnik v4.2.1 and unfork
  • kartotherian/tilelive-bridge => Unforked
  • kartotherian/tilelive-vector => forked due to custom changes making getTile async
  • kartotherian/abaculus => forked due to custom fixes
  • wikimedia/makizushi => forked due to a custom build icon set
  • wikimedia/geojson-mapnikify => forked depends on wikimedia/makizushi
  • wikimedia/tilelive-overlay => forked depends on wikimedia/geojson-mapnikify

One third-party dependency incompatible to mapnik v3.1.x, seems to be mapnik-reference. But de-facto we're don't execute code that's using it. See notes in T327396: Migrate Kartotherian to node-mapnik v4.2.1 and unfork

Open questions

Is that going to be a straight forward upgrade for our dependencies? Some libraries we use are outdated and upgrade them might bring incompatibilities, this will need more tests when mapnik v3.1.x is available.

Acceptance criteria

  • Backport to Debian stretch mapnik v3.1.x when released
  • Wait for mapnik-reference to be updated on upstream Seems that we do not reach that code anywhere.
  • Upgrade node-mapnik in all dependencies
  • Test and deploy

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
MSantos renamed this task from Migrate Kartotherian/Tilerator to Node 10 to [EPIC] Migrate Kartotherian/Tilerator to Node 10.Jan 7 2019, 3:45 PM
Jhernandez renamed this task from [EPIC] Migrate Kartotherian/Tilerator to Node 10 to Migrate Kartotherian/Tilerator to Node 10.Jan 7 2019, 5:16 PM
Jhernandez triaged this task as Medium priority.
Jhernandez added a project: Epic.
MSantos changed the task status from Open to Stalled.Feb 7 2019, 4:06 PM

@MSantos and @Jhernandez , would it perhaps make things far simpler to migrate to use Kubernetes/Docker? Each maps service will be able to use any version of Node, or any native package, or even a different version/distribution of Linux as needed, and it will not affect any other services even if they are running on the same physical hardware. Even migration will be easier, because you will be able to use different version of Node on the same machine by different instances of Tilerator/Kartotherian.

P.S. This approach will also allow Mapnik not to be backported because it can simply be used with an Ubunutu base image - thus simplifying managing of all dependencies.

@Gehel and @Pchelolo any thoughts on Yuri's comment?

Maps will go on k8s eventually afaik, but it's an independent issue. Node 6 support ends in April even for security updates and that is the primary reason for the move.

As said by @Pchelolo, moving to k8s would be great for a number of reason and will happen at some point. But it is a different issue. Also, k8s in the context of WMF will not mean anything can be deployed.

@Gehel is there a doc/procedure describing limits to docker deployment? E.g. would WMF have a docker repository and a well established build process that uploads to it, does it limit to the base images like "must use base Debian but not base Alpine", etc.

My understanding of Docker deployment was "to throw things over the wall" -- developers can build a docker image, throw it to dev-ops, who can deploy it to production in the matter of hours rather than months - simply because all dependencies are baked into the built image, and because dev-ops can compartmentalize different services more easily.

@Gehel is there a doc/procedure describing limits to docker deployment? E.g. would WMF have a docker repository and a well established build process that uploads to it, does it limit to the base images like "must use base Debian but not base Alpine", etc.

Let's move this conversation somewhere else, this is unrelated to the current ticket. Ping me on IRC tomorrow and we'll get started.

@Pchelolo @Gehel I agree that Node 8 upgrade is urgent and must happen. This ticket was discussing Node 10, and mentioned that it was blocked on some dependencies, so I was proposing how to help with the difficulties of dependency upgrades, and k8s might be a good solution to that. I will be happy to chat on IRC. Thanks!!!

Sturm edited projects, added Maps; removed Maps (Kartotherian).
Sturm edited projects, added Maps (Kartographer); removed Maps.
Sturm moved this task from Kartographer to Kartotherian on the Maps board.
Sturm edited projects, added Maps (Kartotherian); removed Maps (Kartographer).

@MSantos : The service::node class which is used by Kartotherian/Tilerator recently gained an option $use_nodejs10, you can extend the kartotherian/tilerator Puppet classes with a new parameter for node10 which then gets passed down to the service::node class and then enable it in deployment-prep. See modules/aqs/manifests/init.pp which already does that.

MSantos renamed this task from Migrate Kartotherian/Tilerator to Node 10 to Migrate Kartotherian/Tilerator to Node 10 when Mapnik is released to v3.1.x.May 1 2019, 8:37 PM
MSantos renamed this task from Migrate Kartotherian/Tilerator to Node 10 when Mapnik is released to v3.1.x to Migrate Kartotherian/Tilerator to Mapnik to v3.1.x when released.
MSantos lowered the priority of this task from Medium to Lowest.
MSantos renamed this task from Migrate Kartotherian/Tilerator to Mapnik to v3.1.x when released to Migrate Kartotherian/Tilerator to Mapnik v3.1.x when released.May 1 2019, 8:43 PM
MSantos updated the task description. (Show Details)
awight added a subscriber: awight.

Unlinking from the parent since this is no longer a blocker. It's still worthwhile to do this maintenance, of course, so we aren't keeping the zombie fork open.

awight renamed this task from Migrate Kartotherian/Tilerator to Mapnik v3.1.x when released to Migrate Kartotherian to Mapnik v3.1.x when released.Oct 27 2022, 10:57 AM
awight renamed this task from Migrate Kartotherian to Mapnik v3.1.x when released to Migrate Kartotherian to libmapnik 3.1.x.
awight changed the task status from Stalled to Open.
awight raised the priority of this task from Lowest to Needs Triage.
awight updated the task description. (Show Details)

Change 851088 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/services/kartotherian@master] [WIP] Update modules to support mapnik3.1

https://gerrit.wikimedia.org/r/851088

Change 851734 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/services/kartotherian@master] Update some node packages

https://gerrit.wikimedia.org/r/851734

Change 851734 merged by jenkins-bot:

[mediawiki/services/kartotherian@master] Update some node packages

https://gerrit.wikimedia.org/r/851734

Change 852870 had a related patch set uploaded (by Awight; author: WMDE-Fisch):

[mediawiki/services/kartotherian@mapnik-3.1] Update modules to support mapnik-3.1

https://gerrit.wikimedia.org/r/852870

Change 852870 merged by Awight:

[mediawiki/services/kartotherian@mapnik-3.1] Update modules to support mapnik-3.1

https://gerrit.wikimedia.org/r/852870

Change 851088 abandoned by WMDE-Fisch:

[mediawiki/services/kartotherian@master] [WIP] Update modules to support mapnik3.1

Reason:

Cherry picked to the feature branch for mapnik3.1

https://gerrit.wikimedia.org/r/851088

Change 853236 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/services/kartotherian@mapnik-3.1] Disable strict mode on vector lib

https://gerrit.wikimedia.org/r/853236

WMDE-Fisch added a project: Epic.
WMDE-Fisch added a subscriber: WMDE-Fisch.

Change 853236 merged by jenkins-bot:

[mediawiki/services/kartotherian@mapnik-3.1] Don't load maps in strict mode in tilelive-vector

https://gerrit.wikimedia.org/r/853236

WMDE-Fisch renamed this task from Migrate Kartotherian to libmapnik 3.1.x to Migrate Kartotherian to mapnik 3.1.x and node-mapnik 4.x.Nov 9 2022, 1:57 PM
WMDE-Fisch added a subscriber: Jgiannelos.

Change 879592 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/services/kartotherian@master] [DNM] Bump node-mapnik to support libmapnik v3.1

https://gerrit.wikimedia.org/r/879592

WMDE-Fisch renamed this task from Migrate Kartotherian to mapnik 3.1.x and node-mapnik 4.x to Migrate Kartotherian to node-mapnik >v4.2.1 to support libmapnik >3.1.x .Jan 19 2023, 1:19 PM
WMDE-Fisch updated the task description. (Show Details)