After our initial end to end tests using EventPlatfom for maps.tile_change events the following considerations came up:
- Currently maps.tile_change topic has only 1 partition and we are planning to have multiple consumers in our consumer group for tile pregeneration. Does it make sense to increase the number of partitions to around the level of concurrency we need for the pregeneration workers?
- The initial design for the maps.tile_change event schema was based in the idea that each tile change was going to be a single event. It turns out that once a day we would have millions of tile changes which as an information is not really valuable to have it to a single event. Maybe changing the schema to bundle multiple tile changes per event would be useful. The reasons are:
- In our tile pregeneration setup we need to receive N amount of events to populate a single task for the pregeneration of N tiles. With a new schema we could just fetch a single message to populate the tile list per task
- In our OpenStreetMap import pipeline we use eventgate to publish events. Using multiple tile changes per event would reduce the amount of HTTP requests we send and the time needed to finish the data import.
- Its not expected to have a need for such granular information when it comes to map tile changes