Page MenuHomePhabricator

Skip no-data tile processing on planet re-generation
Open, LowPublic

Description

Background information

During OS upgrade, the map stack demanded re-generation of tiles from all zoom levels. Inspecting the metrics provided by Tilerator, it's possible to see that several tiles that were processed didn't have any data and were discarded. Tiles are stored considering 4 fields: zoom (int), block (int), idx (int) and tile (blob). Tilerator uses the idx field to keep track of processed tiles.

What

Full planet tile generation is slow and costs a lot, if we have the information about which tiles have data to be generated before the process starts we can generate only the tiles that matter and regenerate the full planet faster. This can benefit two scenarios:

  • Switching between styles.
  • OS and/or Database upgrades.

How

  1. Keep an idx whitelist of tiles that contains data to generate vector-tiles. When generating tiles iterate only over the whitelisted tiles and skip others.
  2. In the background watch for non-whitelisted tiles and update the whitelist if necessary.

Open questions

  • What should be the criteria for tiles to be whitelisted?
  • Should we do this only with the higher zoom levels? Zooms 13, 14, and 15 represent 98.44% of the processed tiles.
  • What kind of storage we will use for the whitelist engine? Simple k/v storage is enough?
  • ... ?

Acceptance criteria

  • ...
  • ...
  • ...

Event Timeline

MSantos created this task.Dec 20 2018, 12:25 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 20 2018, 12:25 AM
Jhernandez renamed this task from [EPIC] Skip no-data tile processing on planet re-generation to Skip no-data tile processing on planet re-generation.Jan 7 2019, 4:10 PM
Jhernandez triaged this task as Low priority.
Jhernandez added a project: Epic.
MSantos moved this task from Backlog to Tile storage on the Maps (Maps-data) board.Apr 1 2020, 1:22 PM