Change Details

==Problem statement== Tegola-vector-tiles service stores rendered vector map tiles as pbf on thanos-swift swift storage. Currently, we store all tiles, and never delete them, thus our swift containers (currently tegola-swift-eqiad-v002 and tegola-swift-codfw-v002) constantly grow. Large containers on swift may cause problems in the swift infrastructure outside the scope of maps because of the way storage is allocated and the lack of system level sharding. We need to work on a way to expire tiles from swift storage, based on when they were accessed last. ===Ideas=== **Tile expiration** * Expire tiles based on last access strategy (LRU kind of thing) ** https://docs.openstack.org/ocata/user-guide/cli-swift-set-object-expiration.html * Create a tegola plugin which will add a related header (any of `X-Delete-At` or `X-Delete-After`) ** Either create a small http proxy to do so ** Or use something off the shelf * Tiles we are regenerating via crons jobs should not affect last access timestamp **Cache sharding** * Allow tegola to use multiple cache backends ** Extend tegola to use multiple cache backends with sharding ** Write a proxy to do the sharding between tegola and multiple swift containers

==Problem statement== [[ https://wikitech.wikimedia.org/wiki/Maps/v2/Architecture | Tegola-vector-tiles ]]service stores rendered vector map tiles as pbf on thanos-swift swift storage. Currently, we store all tiles, and never delete them, thus our swift containers (currently tegola-swift-eqiad-v002 and tegola-swift-codfw-v002) constantly grow. Large containers on swift may cause problems in the swift infrastructure outside the scope of maps because of the way storage is allocated and the lack of system level sharding. We need to work on a way to expire tiles from swift storage, based on when they were accessed last. ===Ideas=== **Tile expiration** * Expire tiles based on last access strategy (LRU kind of thing) ** https://docs.openstack.org/ocata/user-guide/cli-swift-set-object-expiration.html * Create a tegola plugin which will add a related header (any of `X-Delete-At` or `X-Delete-After`) ** Either create a small http proxy to do so ** Or use something off the shelf * Tiles we are regenerating via crons jobs should not affect last access timestamp **Cache sharding** * Allow tegola to use multiple cache backends ** Extend tegola to use multiple cache backends with sharding ** Write a proxy to do the sharding between tegola and multiple swift containers

==Problem statement== [[ https://wikitech.wikimedia.org/wiki/Maps/v2/Architecture | Tegola-vector-tiles ]]service stores rendered vector map tiles as pbf on thanos-swift swift storage. Currently, we store all tiles, and never delete them, thus our swift containers (currently tegola-swift-eqiad-v002 and tegola-swift-codfw-v002) constantly grow. Large containers on swift may cause problems in the swift infrastructure outside the scope of maps because of the way storage is allocated and the lack of system level sharding. We need to work on a way to expire tiles from swift storage, based on when they were accessed last. ===Ideas=== **Tile expiration** * Expire tiles based on last access strategy (LRU kind of thing) ** https://docs.openstack.org/ocata/user-guide/cli-swift-set-object-expiration.html * Create a tegola plugin which will add a related header (any of `X-Delete-At` or `X-Delete-After`) ** Either create a small http proxy to do so ** Or use something off the shelf * Tiles we are regenerating via crons jobs should not affect last access timestamp **Cache sharding** * Allow tegola to use multiple cache backends ** Extend tegola to use multiple cache backends with sharding ** Write a proxy to do the sharding between tegola and multiple swift containers