We currently heavily rely on tile pre-generation in our map request flow.
Tegola allows cache seeding which can be used in a similar manner with our current pre-generation architecture.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | ssastry | T263854 [Maps] Modernize Vector Tile Infrastructure | |||
| Resolved | MSantos | T269690 View an Example Map (using Tegola vector tile server) | |||
| Resolved | MSantos | T270169 Bootstrap Tegola vector-tile server with baseline MVT schema from OSM bright | |||
| Resolved | Jgiannelos | T270175 Support vector tile pre-generation | |||
| Resolved | Jgiannelos | T289771 Add kafka support for tile-pregeneration events | |||
| Resolved | Jgiannelos | T290982 Support expired tile deduplication | |||
| Resolved | Jgiannelos | T293366 Performance considerations about the current usage of EventPlatform from maps | |||
| Resolved | Jgiannelos | T294011 Sending events to `maps.tiles_change` stream is failing |
Here is the relevant patch for k8s deployment cronjob:
https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/701938
Change 712375 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[operations/software/tegola@wmf/v0.14.x] scripts: Fix reading default value for env var
Change 712375 merged by jenkins-bot:
[operations/software/tegola@wmf/v0.14.x] scripts: Fix reading default value for env var
Change 713904 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[operations/puppet@production] maps: Install kafkacat on maps masters
Change 713904 merged by Hnowlan:
[operations/puppet@production] maps: Install kafkacat on osm master nodes
On tegola staging (k8s) we managed to have a couple of succesful runs of the cronjob defined here: https://github.com/wikimedia/operations-software-tegola/blob/wmf/v0.14.x/scripts/pregenerate-maps-tiles.sh
There is a parsing issue for input with format:
{"tile": "15/11987/17639"}
{"tile": "15/8350/12771"}
{"tile": "15/24032/11194"}
{"tile": "15/15532/10901"}
{"tile": "15/17662/11303"}
{"tile": "15/23112/12439"}
{"tile": "15/22702/12313"}
{"tile": "15/15802/10706"}
{"tile": "15/23433/12278"}
{"tile": "15/22868/12148"}Tegola raised this error:
2021-08-20 14:06:18 [INFO] root.go:62: Loading config file: /etc/tegola/config.toml
2021-08-20 14:06:18 [INFO] config.go:306: loading local config (/etc/tegola/config.toml)
2021-08-20 14:06:18 [INFO] providers.go:82: registering provider(type): osm (mvt_postgis)
2021-08-20 14:06:19 [INFO] tile_list.go:91: zoom list: []
2021-08-20 14:06:19 [INFO] cache.go:204: waiting for workers to finish up
2021-08-20 14:06:19 [INFO] cache.go:214: all workers are done
2021-08-20 14:06:19 [INFO] provider.go:256: cleaning up providers
2021/08/20 14:06:19 postgis.go:853: cleaning up postgis providers
Error: error seeding tile ({Z:15 X:22702 Y:12313}): ERROR: syntax error at or near ")" (SQLSTATE 42601)
error seeding tile ({Z:15 X:22702 Y:12313}): ERROR: syntax error at or near ")" (SQLSTATE 42601)
Usage:
tegola cache seed tile-list filename|- [flags]
Examples:
tile-list my-tile-list.txt
Flags:
--format string 4 character string where the first character is a non-numeric delimiter followed by 'z', 'x' and 'y' defining the coordinate order (default "/zxy")
-h, --help help for tile-list
--max-zoom uint max zoom to seed cache to
--min-zoom uint min zoom to seed cache from
Global Flags:
--concurrency int the amount of concurrency to use. defaults to the number of CPUs on the machine (default 40)
--config string path to config file (default "config.toml")
--map string map name as defined in the config
--overwrite overwrite the cache if a tile already exists (default false)Hopefully this is going to be easily reproducible on the local setup to avoid the complexity of trying to debug the jobs on k8s.
Change 722825 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[operations/puppet@production] Add script to send tile invalidation events
Change 726522 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[operations/software/tegola@wmf/v0.14.x] tile-pregeneration: Exclude canary test events
Change 726522 merged by jenkins-bot:
[operations/software/tegola@wmf/v0.14.x] tile-pregeneration: Exclude canary test events
Change 722825 merged by Hnowlan:
[operations/puppet@production] maps: Add script to send tile invalidation events
Change 737898 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[operations/puppet@production] maps: Add cronjob to send tile invalidation events
Currently we only have a way to send batched tiles but because of eventgate triggering a single tile is a matter of a curl command based on the template json we already maintain in the maps masters.
Change 737898 merged by Hnowlan:
[operations/puppet@production] maps: Add cronjob to send tile invalidation events