Change Details

= Background information Evaluate the WMF OSM pipeline and create a report for possible improvements. The topics are the following: - [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/644482 | Current usage of imposm3 ]], see {T269582} - [[ https://github.com/thesocialdev/kartosm/blob/master/mapping.yml | Imposm3 mapping ]] - It’s worth mentioning that the imposm3 mapping was intentionally created to match the osm2pgsql schema, for backward compatibility without having to change the .tm2 source queries. - [[ https://gerrit.wikimedia.org/r/plugins/gitiles/maps/spikes/+/refs/heads/master/config/tegola-kartotherian-queries.toml | Tegola queries for client-side rendering ]] - This .toml configuration was created based on osm-bright.tm2source AR: I have some ideas on how to improve performance, reduce tile size and simplify maintenance. A few quick suggestions: Use database views. This will allow DBAs to improve the queries without needing to redeploy tegola. If Postgres 12.x is available we could migrate to using ST_AsMVT Depending on the update interval required, materialized views can be used to precompute simplification. Possible improvements for the current schema to integrate legacy code with Tegola MS: We are still going to keep part of the infrastructure alive for a while, but we plan to replace our current vector-tile strategy with Tegola. By integrating our mapnik based nodejs service with Tegola, what kind of improvements in the current DB schema we might need to expect? = Open questions - We also have a nodejs endpoint that retrieves GeoJSON for OSM relations that matches specific Wikidata entities. This endpoint uses the same OSM database, based on osm2pgsql DB schema. Are there any concerns about this service sharing the same DB? - We are evaluating OpenMapTiles as a schema for our tiles strategy - Are there any performance/architecture problems that you can identify compared to our current schema? - Is this a viable option given our scale? - In our production systems we use Debian buster which offers packages for Postgres v11.x / PostGIS v2.5x. According to the ST_AsMVT docs, this feature is supported on PostGIS v2.5x. Is there a specific functionality that we want to use that is available in a newer version or are we OK with the one provided by the Debian repositories? = Acceptance criteria - [] Documentation is updated - [] Follow-up tasks are created to improve our OSM database - [x] imposm3 mapping - [x] Tegola queries (simplified using pg views/functions)

= Background information Evaluate the WMF OSM pipeline and create a report for possible improvements. The topics are the following: - [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/644482 | Current usage of imposm3 ]], see {T269582} - [[ https://github.com/thesocialdev/kartosm/blob/master/mapping.yml | Imposm3 mapping ]] - It’s worth mentioning that the imposm3 mapping was intentionally created to match the osm2pgsql schema, for backward compatibility without having to change the .tm2 source queries. - [[ https://gerrit.wikimedia.org/r/plugins/gitiles/maps/spikes/+/refs/heads/master/config/tegola-kartotherian-queries.toml | Tegola queries for client-side rendering ]] - This .toml configuration was created based on osm-bright.tm2source AR: I have some ideas on how to improve performance, reduce tile size and simplify maintenance. A few quick suggestions: Use database views. This will allow DBAs to improve the queries without needing to redeploy tegola. If Postgres 12.x is available we could migrate to using ST_AsMVT Depending on the update interval required, materialized views can be used to precompute simplification. Possible improvements for the current schema to integrate legacy code with Tegola MS: We are still going to keep part of the infrastructure alive for a while, but we plan to replace our current vector-tile strategy with Tegola. By integrating our mapnik based nodejs service with Tegola, what kind of improvements in the current DB schema we might need to expect? = Open questions - We also have a nodejs endpoint that retrieves GeoJSON for OSM relations that matches specific Wikidata entities. This endpoint uses the same OSM database, based on osm2pgsql DB schema. Are there any concerns about this service sharing the same DB? - We are evaluating OpenMapTiles as a schema for our tiles strategy - Are there any performance/architecture problems that you can identify compared to our current schema? - Is this a viable option given our scale? - In our production systems we use Debian buster which offers packages for Postgres v11.x / PostGIS v2.5x. According to the ST_AsMVT docs, this feature is supported on PostGIS v2.5x. Is there a specific functionality that we want to use that is available in a newer version or are we OK with the one provided by the Debian repositories? = Acceptance criteria - [] Documentation is updated - [x] Follow-up tasks are created to improve our OSM database - [x] imposm3 mapping - [x] Tegola queries (simplified using pg views/functions)

= Background information Evaluate the WMF OSM pipeline and create a report for possible improvements. The topics are the following: - [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/644482 | Current usage of imposm3 ]], see {T269582} - [[ https://github.com/thesocialdev/kartosm/blob/master/mapping.yml | Imposm3 mapping ]] - It’s worth mentioning that the imposm3 mapping was intentionally created to match the osm2pgsql schema, for backward compatibility without having to change the .tm2 source queries. - [[ https://gerrit.wikimedia.org/r/plugins/gitiles/maps/spikes/+/refs/heads/master/config/tegola-kartotherian-queries.toml | Tegola queries for client-side rendering ]] - This .toml configuration was created based on osm-bright.tm2source AR: I have some ideas on how to improve performance, reduce tile size and simplify maintenance. A few quick suggestions: Use database views. This will allow DBAs to improve the queries without needing to redeploy tegola. If Postgres 12.x is available we could migrate to using ST_AsMVT Depending on the update interval required, materialized views can be used to precompute simplification. Possible improvements for the current schema to integrate legacy code with Tegola MS: We are still going to keep part of the infrastructure alive for a while, but we plan to replace our current vector-tile strategy with Tegola. By integrating our mapnik based nodejs service with Tegola, what kind of improvements in the current DB schema we might need to expect? = Open questions - We also have a nodejs endpoint that retrieves GeoJSON for OSM relations that matches specific Wikidata entities. This endpoint uses the same OSM database, based on osm2pgsql DB schema. Are there any concerns about this service sharing the same DB? - We are evaluating OpenMapTiles as a schema for our tiles strategy - Are there any performance/architecture problems that you can identify compared to our current schema? - Is this a viable option given our scale? - In our production systems we use Debian buster which offers packages for Postgres v11.x / PostGIS v2.5x. According to the ST_AsMVT docs, this feature is supported on PostGIS v2.5x. Is there a specific functionality that we want to use that is available in a newer version or are we OK with the one provided by the Debian repositories? = Acceptance criteria - [] Documentation is updated - [x] Follow-up tasks are created to improve our OSM database - [x] imposm3 mapping - [x] Tegola queries (simplified using pg views/functions)