Thanks Kevin. Do you have a link to the repository yet, so I can start estimating what that will take?

@MaxSem @Yurik Can you point Chris to something?

@csteipp https://github.com/nyurik/kartotherian/tree/dyngeneration

And @Yurik, where are you guys at in the project. Is the architecture finalized for this now? What are your timelines for deploying?

@MoritzMuehlenhoff, are you ok with postgresql here?

@Yurik / @MaxSem, can we schedule some time soon to go over the data flow of the different pieces?

Overall architecture is mostly stabilized. Max is working on SQL queries most of the time, and I am implementing tile generation (internal service), which will be most likely implemented as part of the kartotherian, but ran under different configuration. Kartotherian is mostly done - it serves tiles from the storage (cassandra), after converting them from vector to image form. We could discuss it today - next week is Wikimania, might be a bit crazy with the traveling.
Postgress is the only option for us at the moment -- all OSM stacks from what I know use Postgis extension for all Geo coordinate work (from what I heard, MariaDb and MySql are working on spatial functions as well, so in the future we might want to reconsider Postgres)

In T105051#1445301, @csteipp wrote:

And @Yurik, where are you guys at in the project. Is the architecture finalized for this now? What are your timelines for deploying?

@MoritzMuehlenhoff, are you ok with postgresql here?

Postgres is general is a fine choice; it has good security support record and is well supported upstream and in Debian (and since this is a new service it'll be based on Jessie).

Just had a chat with Chris, giving him an overview.

DFD from my conversation with Max

The diagram is correct, except we call them PBFs, not PBEs. As for user requests, we currently only handle "/<source>/<z>/<x>/<y>.<format>" style requests. Will ignore non-GET requests. Also we handle /static/* -- serving of anything inside the /static dir as raw content (html, js). Lastly, ignore the "tilerator.js" - it will be removed from kartotherian repo, and will go into the separate repository. The tilerator will not be accessible by users.

update: the code has been hugely revised and shortened, and deployed to prod (not accessible yet).

To see it running, set up a tunnel:

ssh -L 8888:localhost:4000 maps-test200{1-4}.codfw.wmnet
browse to http://localhost:8888/static

Also, the service is running at ns512621.ip-167-114-156.net/static via varnish, or on port 4000 without varnish, but we are constantly experimenting with it, could be broken at any moment.

I looked at kartotherian, kartotherian-core, kartotherian-cassandra, and xmldoc. They seem to be reasonable with the current config.

@MaxSem and @Yurik are discussing if opening up on the fly generation should be allowed. Currently, with no customers depending on this service the impact from a DoS would be minimal, so the risk is also very low. However once other tools are using the maps, it seems like a bad idea.

@mobrovac, do you have any concerns about any of the libraries they are using? Here are the ones that I don't think we include in service-runner by default.

"body-parser": "^1.12.3",
"buffertools": "^2.1.2",
"compression": "^1.4.3",
"domino": "^1.0.18",
"express": "^4.12.3",
"leaflet": "~0.7.3",
"mapnik": "~3.4.0",
"minimist": "0.2.*",
"node-uuid": "^1.4.3",
"tilelive": "~5.8.2",
"tilelive-bridge": "~1.4.0",
"tilelive-file": "^0.0.3",
"tilelive-vector": "~3.3.0",
"underscore": "^1.8.3"
"cassandra-driver": "^2.1.1",
"multistream": "^1.6.1",
"promistreamus": "^0.1.0"

@Yurik, I'm assuming if any of these libraries gets a security update, you're watching for those?

The licensing of osm-bright-style seemed a little non-standard, are we sure it's ok to include that?

This license is 3-clause BSD. It's DFSG compatible so I don't see any problems here. Even though 2-clause version would've been nicer indeed, it's a fork, so we didn't get to choose the license.

In T105051#1537874, @csteipp wrote:

@mobrovac, do you have any concerns about any of the libraries they are using? Here are the ones that I don't think we include in service-runner by default.

I will have to take a closer look, but only by looking at the list I can tell that these need some updates. For example, node-uuid was used by the service-template-node, but has since been replaced by cassandra-uuid, a pure-JS UUID library, ported from Cassandra's Node.JS driver and maintained by the Services team. I have opened T109021: Code review of Maps service to address this and similar issues.

In T105051#1537996, @MaxSem wrote:

This license is 3-clause BSD. It's DFSG compatible so I don't see any problems here. Even though 2-clause version would've been nicer indeed, it's a fork, so we didn't get to choose the license.

That it is. Good enough.

@mobrovac, definitely good to use best libraries we can, thanks for pointing out that specific case.

In general though, since these are essentially going unreviewed into production (no way I can review 500 kloc's before this goes out), are you comfortable that these are reputable / industry standard / responsible? I'm basically saying "You are responsible for what you require", and I want to make sure that in your professional option, these are responsible things to pull in. We don't have a specific list of service library requirements yet (on my list of things to do), but basically that we're reasonably confident they don't contain backdoors, or are going to allow some random person to add a backdoor with a pull request, are going to provide security fixes if vulnerabilities are found, etc.

If that's the case, we can go ahead and close this out. If you're concerned about any of these, let me know and I'll take a deeper look.

In T105051#1539973, @csteipp wrote:

@mobrovac, definitely good to use best libraries we can, thanks for pointing out that specific case.

In general though, since these are essentially going unreviewed into production (no way I can review 500 kloc's before this goes out), are you comfortable that these are reputable / industry standard / responsible? I'm basically saying "You are responsible for what you require", and I want to make sure that in your professional option, these are responsible things to pull in. We don't have a specific list of service library requirements yet (on my list of things to do), but basically that we're reasonably confident they don't contain backdoors, or are going to allow some random person to add a backdoor with a pull request, are going to provide security fixes if vulnerabilities are found, etc.

If that's the case, we can go ahead and close this out. If you're concerned about any of these, let me know and I'll take a deeper look.

So, setting aside the update of the service and modules to the latest service-template-node, most of the unknown modules listed here are really maps-specific, so for these I'm throwing the ball back in @Yurik's and @MaxSem's courtyard. As for the rest, I'd need to take a closer look at buffertools, since it's a binary dependency dealing with buffering (and we all know that to be a sweet a spot for exploits). Given that it's Friday afternoon in Europe, could you take a look at that one, @csteipp ? If that's settled, I'm OK with moving forward (keeping in mind @Yurik's promise about updating and mine to code-review the service(s) next week).

In T105051#1537874, @csteipp wrote:

@MaxSem and @Yurik are discussing if opening up on the fly generation should be allowed. Currently, with no customers depending on this service the impact from a DoS would be minimal, so the risk is also very low. However once other tools are using the maps, it seems like a bad idea.

I would expand this and say that it's unacceptable to do that. We usually struggle with exploit vectors we are not aware of, and I don't think we should start accepting known ones. Mind you, my mind could change after I do T109021: Code review of Maps service.