Change Details

After our initial end to end tests using EventPlatfom for `maps.tile_change` events the following considerations came up: * Currently `maps.tile_change` topic has only 1 partition and we are planning to have multiple consumers in our consumer group for tile pregeneration. Does it make sense to increase the number of partitions to around the number of pregeneration workers? * The initial design for the `maps.tile_change` [[ https://schema.wikimedia.org/repositories/primary/jsonschema/maps/tile_change/current.yaml | event schema ]] was based in the idea that each tile change was going to be a single event. It turns out that once a day we would have millions of tile changes but as an information its not really valuable to have it to a single event. Maybe changing the schema to bundle multiple tile changes per event would be useful. The reasons are: * In our tile pregeneration setup we need to receive N amount of events to populate a single task for the pregeneration of N tiles. With a new schema we could just fetch a single message on populate the tile list per task * In our OpenStreetMap import pipeline we use eventgate to publish events. Using multiple tile changes per event would reduce the amount of HTTP requests we send and the time needed to finish the data import. * Its not expected to have a need for such granular information when it comes to map tile changes

After our initial end to end tests using EventPlatfom for `maps.tile_change` events the following considerations came up: * Currently `maps.tile_change` topic has only 1 partition and we are planning to have multiple consumers in our consumer group for tile pregeneration. Does it make sense to increase the number of partitions to around the number of pregeneration workers? * The initial design for the `maps.tile_change` [[ https://schema.wikimedia.org/repositories/primary/jsonschema/maps/tile_change/current.yaml | event schema ]] was based in the idea that each tile change was going to be a single event. It turns out that once a day we would have millions of tile changes which as an information is not really valuable to have it to a single event. Maybe changing the schema to bundle multiple tile changes per event would be useful. The reasons are: * In our tile pregeneration setup we need to receive N amount of events to populate a single task for the pregeneration of N tiles. With a new schema we could just fetch a single message on populate the tile list per task * In our OpenStreetMap import pipeline we use eventgate to publish events. Using multiple tile changes per event would reduce the amount of HTTP requests we send and the time needed to finish the data import. * Its not expected to have a need for such granular information when it comes to map tile changes

After our initial end to end tests using EventPlatfom for `maps.tile_change` events the following considerations came up: * Currently `maps.tile_change` topic has only 1 partition and we are planning to have multiple consumers in our consumer group for tile pregeneration. Does it make sense to increase the number of partitions to around the number of pregeneration workers? * The initial design for the `maps.tile_change` [[ https://schema.wikimedia.org/repositories/primary/jsonschema/maps/tile_change/current.yaml | event schema ]] was based in the idea that each tile change was going to be a single event. It turns out that once a day we would have millions of tile changes butwhich as an information itss not really valuable to have it to a single event. Maybe changing the schema to bundle multiple tile changes per event would be useful. The reasons are: * In our tile pregeneration setup we need to receive N amount of events to populate a single task for the pregeneration of N tiles. With a new schema we could just fetch a single message on populate the tile list per task * In our OpenStreetMap import pipeline we use eventgate to publish events. Using multiple tile changes per event would reduce the amount of HTTP requests we send and the time needed to finish the data import. * Its not expected to have a need for such granular information when it comes to map tile changes