Page MenuHomePhabricator

Deploy meddo / cleartables as part of tilerator / kartotherian
Closed, ResolvedPublic

Description

The lifecycle of meddo and its configuration is strongly linked to tilerator. It would make a lot of sense to deploy it as part of the tilerator-deploy repo, probably as a submodule.

@MaxSem you understand the build / deployment process better than I do, could you confirm this actually make sense?

Event Timeline

I see two ways of deploying: either as a submodule, or as part of docker built script.

  • The submodule is fairly straightforward, because kartotherian itself is already deployed as one: https://phabricator.wikimedia.org/diffusion/GMKD/browse/master/ -- the src/ dir is a submodule. You can create a few more top-level submodules - tm2, tm2source, fonts.
  • The dockerfile is a bit tricker - the dockerfile is autogenerated from the package.json, so adding things to it is not very straightforward, especially when we want to keep the plugins (e.g. meddo) separate from the core kartotherian.

Change 348045 had a related patch set uploaded (by Gehel):
[maps/tilerator/deploy@master] adding meddo as a submodule of tilerator

https://gerrit.wikimedia.org/r/348045

After discussion with @Pnorman the nicest solution we can come up with is:

  • deploy those dependencies as npm modules of tilerator and kartotherian
  • since those are not actually requirements for tilerator / kartotherian but only of our specific implementation:
    • replace the package.json symlink in our deploy repositories with a real file
    • have that deploy/package.json depend on tilerator / kartotherian to drag all their dependencies
    • add our specific additional dependencies to the deploy/package.json

The above can be implemented in multiple steps, since some cleaning is required, and compatibility with the current vector tiles is needed.

  1. replace the symlink with a real file
  2. reduce the deploy/package.json to a dependency on tilerator / kartotherian
  3. add our own dependencies

Hi @MaxSem - can you take a look and confirm that the steps below look good, please?

After discussion with @Pnorman the nicest solution we can come up with is:

  • deploy those dependencies as npm modules of tilerator and kartotherian
  • since those are not actually requirements for tilerator / kartotherian but only of our specific implementation:
    • replace the package.json symlink in our deploy repositories with a real file
    • have that deploy/package.json depend on tilerator / kartotherian to drag all their dependencies
    • add our specific additional dependencies to the deploy/package.json

The above can be implemented in multiple steps, since some cleaning is required, and compatibility with the current vector tiles is needed.

  1. replace the symlink with a real file
  2. reduce the deploy/package.json to a dependency on tilerator / kartotherian
  3. add our own dependencies

Note: replacing the symlinked package.json by an actual file will require manual change that file when we upgrade kartotherian / tilerator

Change 386410 had a related patch set uploaded (by Gehel; owner: Gehel):
[maps/kartotherian/deploy@master] Replace package.json symlink by actual file with same content

https://gerrit.wikimedia.org/r/386410

Change 386411 had a related patch set uploaded (by Gehel; owner: Gehel):
[maps/tilerator/deploy@master] Replace package.json symlink by actual file with same content

https://gerrit.wikimedia.org/r/386411

Some more context (taken from my last comment on the associated gerrit CR):

The basic idea is that we have dependencies that need to be bundled with kartotherian at deployment but which do not make sense to be direct dependencies of kartotherian.

In this specific case, we need to add our map style to the deployment, but it does not make sense to add it a kartotherian dependencies, because other users or kartotherian will use different styles.

Using npm to manage those dependencies seem to make sense, as they are runtime dependencies of kartotherian / tilerator. There might be a better way, and we are very much open to suggestions!

@mobrovac, @Eevans: your input on this would be welcomed. I'll create a few more CR to expose the implementation we are thinking about, but feel free to tell me there is a better way to do it!

Change 386584 had a related patch set uploaded (by Gehel; owner: Gehel):
[maps/kartotherian/deploy@master] Replace direct dependencies with kartotherian dependency.

https://gerrit.wikimedia.org/r/386584

Change 386585 had a related patch set uploaded (by Gehel; owner: Gehel):
[maps/kartotherian/deploy@master] Add meddo and ClearTables as deploy dependencies.

https://gerrit.wikimedia.org/r/386585

The 2 patches above expose the implementation of our initial discussion with @Pnorman. They are probably wrong, ugly and missing pieces, but they can be used as a base for a concrete discussion.

@mobrovac: I hope this makes our intent a bit clearer.

Side note: I'm wondering (meaning that this is maybe a terrible idea, let me know) if it wouldn't make sense to have an intermediate packaging repo. Basically, the composing that is done in the deploy repo could be moved to this packaging repo. The packaging repo would depend on kartotherian and meddo. This would allow to keep the same structure in the deploy repo (a symlink to a package.json file), while providing a way to define deploy dependencies. On the other hand, this also adds yet another repository and a bunch of inherent complexity and additional steps for deployment.

What we could do here is add a field to our deploy field in package.json that could list extra node module dependencies to be installed during the build-repo step. Something like:

"deploy": {
  "node": "6.11.1",
  // other directives
  "modules": {
    "node_mod_1": "^1.2.3",
    "node_mod_2": "2.3.4"
  }
}

Then, these modules would be installed alongside other modules in the deploy repo during the build process.

Change 394027 had a related patch set uploaded (by Gehel; owner: Guillaume Lederrey):
[maps/kartotherian/package@master] align package.json with currently deployed version

https://gerrit.wikimedia.org/r/394027

Change 394027 merged by Gehel:
[maps/kartotherian/package@master] align package.json with currently deployed version

https://gerrit.wikimedia.org/r/394027

Change 394575 had a related patch set uploaded (by Gehel; owner: Gehel):
[maps/kartotherian/deploy@master] New packaging repository for kartotherian

https://gerrit.wikimedia.org/r/394575

Change 394575 merged by Gehel:
[maps/kartotherian/deploy@master] New packaging repository for kartotherian

https://gerrit.wikimedia.org/r/394575

Gehel renamed this task from Deploy meddo as part of tilerator-deploy to Deploy meddo as part of tilerator / kartotherian.Dec 1 2017, 3:21 PM

Change 395543 had a related patch set uploaded (by Gehel; owner: Gehel):
[maps/tilerator/deploy@master] New packaging repository for tilerator

https://gerrit.wikimedia.org/r/395543

Change 395543 merged by Gehel:
[maps/tilerator/deploy@master] New packaging repository for tilerator

https://gerrit.wikimedia.org/r/395543

kartotherian has a new working packaging repo. tilerator is in progress. Still to do:

  • publish and use the new version of kartotherian/snapshot
  • deploy both tilerator and kartotherian with new packaging repo
  • add the meddo / cleartable dependencies once we confirm the new packaging repo are working fine in production
Gehel renamed this task from Deploy meddo as part of tilerator / kartotherian to Deploy meddo / cleartables as part of tilerator / kartotherian.Dec 5 2017, 5:23 PM

Change 397836 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: extract style used by kartotherian / tilerator as parameter

https://gerrit.wikimedia.org/r/397836

Change 397836 merged by Gehel:
[operations/puppet@production] maps: extract style used by kartotherian / tilerator as parameter

https://gerrit.wikimedia.org/r/397836

Change 397846 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: update style to match latest kartotherian version

https://gerrit.wikimedia.org/r/397846

Change 397846 merged by Gehel:
[operations/puppet@production] maps: update style to match latest kartotherian version

https://gerrit.wikimedia.org/r/397846

Change 386410 abandoned by Gehel:
Replace package.json symlink by actual file with same content

https://gerrit.wikimedia.org/r/386410

Change 386411 abandoned by Gehel:
Replace package.json symlink by actual file with same content

https://gerrit.wikimedia.org/r/386411

Change 348045 abandoned by Gehel:
adding meddo and ClearTables as a submodule of tilerator

https://gerrit.wikimedia.org/r/348045

Current status:

both tilerator and kartotherian now have an intermediate "packaging" repository, which allow us to aggregate resources which are not directly part of either tilerator or kartotherian:

Dependencies that are not available as Debian packages but need to be present on the maps servers should be deployed with this mechanism. This is the case for meddo and cleartables.

@Pnorman can you integrate meddo / cleartables (and anything else that you know we need)?

Thanks!

Change 386584 abandoned by Gehel:
Replace direct dependencies with kartotherian dependency.

https://gerrit.wikimedia.org/r/386584

Change 386585 abandoned by Gehel:
Add meddo as deploy dependencies.

https://gerrit.wikimedia.org/r/386585

Pnorman claimed this task.

This was done