Page MenuHomePhabricator

Deploying new vector tiles to production
Closed, DeclinedPublic

Description

With the work progressing, it's worth considering the deploying to production aspects, even if we don't know what the future of the team and maps will be. We want to avoid a flag day and any points of no return where we don't have a backout strategy.

Additional components to build

Both ClearTables and meddo require building. ClearTables needs python and python-yaml to build, which are both required by other components, and meddo requires some vector tile design software (e.g. kosmtik).

OSM database

The new work requires an osm2pgsql database loaded with the ClearTables osm2pgsql style. This is imported and updated with osm2pgsql, the same tool we already use, just different build. It requires about the same space as currently used (~650gb est).

The best way to deploy this would be to request a new servers, import to it, and everything is working, release the old ones to the hardware pool. If our operations setup can't do this, then we need to either import on the same machines using spare space, or take one of the existing machines, pull it out of production, and import on it.

An import will take 24-72 hours on reasonable hardware.

ClearTables is versioned, and anything we do will be based on a released version

Geoshapes

Right now my understanding is geoshapes is querying the rendering database. If so, we'll need to keep an osm2pgsql rendering database imported with the pgsql style around, but won't need the same capacity.

Other data

External shapefiles

These are loaded with a python script, and updated with the same script. It takes a few minutes to load, and a few minutes to update if the data has changed. The updating can be done while rendering is being done.

Borders

TBD - this involves running osmborder

Kartotherian

We can run the db -> vector and vector -> raster processes on the same machines by adding a second style to the Kartotherian config. The backout here is reverting the config change

Cached vector tiles

We will need to pre-render at least low zooms, so need vector tile storage of similar capacity to what we have right now.

Kartotherian

We should be able to make the raster tiles available on a different URL, and then test them. When ready, we can swap in the new name for the old name.

Related Objects

StatusSubtypeAssignedTask
OpenNone
InvalidNone
OpenFeatureNone
InvalidNone
OpenNone
ResolvedPnorman
DeclinedNone
DeclinedGehel
Resolved Mholloway
ResolvedPnorman
ResolvedGehel
ResolvedPnorman
ResolvedGehel
ResolvedGehel
ResolvedGehel
ResolvedNone
DeclinedNone
ResolvedPnorman
ResolvedNone
ResolvedNone
Resolved mobrovac
ResolvedGehel
ResolvedPnorman
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
ResolvedGehel
ResolvedSBisson
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedGehel
ResolvedGehel
ResolvedGehel
ResolvedPnorman
Resolved Mholloway
ResolvedGehel

Event Timeline

Per discussion, we have four maps servers, each of which has all the components on them. The only resource constraints are disk space, and this is insufficient to have enough space for both the current db + tiles and the new db + tiles. Therefor, we'll go with taking an existing machine out of the pool, loading it up with the new stuff, testing, doing a second machine, testing, switching over to the newly loaded machines, testing under full load, then switch the last two machines. This way, at no point are we in a situation where one machine failure could bring down the cluster.

It occurs to me that we haven't really discussed the geoshapes/geolines and what those do. Do those hit the rendering databases?

@Pnorman geoshape service accesses both the line and polygon tables. If we can generate an alternative data source for shapes, it would be good (because we could also solve the bug with non-closed relations like roads and rivers)

It has been decided that because we don't have a dedicated user-focused product manager for maps changes we can't make material product changes for maps, so the new styles work will not be deployed.