Page MenuHomePhabricator

Externalize tile storage for maps
Closed, ResolvedPublic

Description

At the moment, maps servers host a geo database (postgresql) with an import of OSM data, an object store (cassandra) which stores the pregenerated vector tiles, the process to generate vector tiles (tilerator) and the process to serve raster tiles to the clients (kartotherian). The object store is very generic and is basically just a key-value store, not using any advanced feature of cassandra.

Since the object store has mostly nothing specific to maps, it would make sense to externalize it to a shared object store. The obvious candidates would be cassandra as used by restbase, swift, or a custom object store (@Eevans has some thoughts about this, I'll let him elaborate).

Any of those solution is likely to require some changes to tilerator / kartotherian, so it is likely to require some development time (and it is unclear if / when we can get access to dev resources for maps).

Some of the question we want to ask before getting further into this discussion:

  • number of tiles needing to be stored
  • overall storage size
  • access patterns to the tiles (number of read / writes per seconds, taking into account that writes are async)
  • access times expected from the storage
  • ...

Event Timeline

Gehel mentioned this in Unknown Object (Task).Jun 5 2018, 3:10 PM

Any of those solution is likely to require some changes to tilerator / kartotherian, so it is likely to require some development time (and it is unclear if / when we can get access to dev resources for maps).

It'll need development time, but shouldn't need *too* much, since we're not doing anything fancy with Cassandra.

Joe triaged this task as Medium priority.Jun 18 2018, 12:44 PM
Vvjjkkii renamed this task from Externalize tile storage for maps to plbaaaaaaa.Jul 1 2018, 1:05 AM
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.

Since we already have done some research for using swift as a vector tile storage and tegola is already running backed by swift on staging k8s should we close this ticket or there is more things we need to investigate?

Tegola staging is already using swift for some time already as a tile storage backend and metrics look OK.

Jgiannelos claimed this task.