Change Details

Currently, we host flamegraph SVGZ and compressed PHP Excimer logs as an open directory from a single host per-dc from performance.wm.o. We replicate these artifacts to Swift with a 3-year TTL set on them. The hosts backing this open directory have smaller disks than what we keep, requiring us to manually manage the artifacts which can balloon unexpectedly under periods of extra load. To support the possibility of making this flamegraph-generating process less disk-dependent (and possibly make it k8s-ready), I propose we serve these artifacts directly from swift. However, it'd be nice to bring on some tool so we do not have to write a webapp to generate the directory lists in house. I'm imagining something like Apache's DirectoryListings rendered html list of the objects in swift and proxy users to download the object from object storage directly. @MatthewVernon indicated that hosting a page like this with S3 might be easy. I've no preference on object store flavor, but with slight edge to S3 as it has wider support among ETL processors I've looked at thus far. TL;DR - Does Data Persistence support anything that we could proxy to that could serve as an "open directory" listing of artifacts stored in and served from a bucket?

Currently, we host flamegraph SVGZ and compressed PHP Excimer logs as an web-accessible directory from a single host per-dc from performance.wm.o. We replicate these artifacts to Swift with a 3-year TTL set on them. The hosts backing this directory have smaller disks than what we keep, requiring us to manually manage the artifacts which can balloon unexpectedly under periods of extra load. To support the possibility of making this flamegraph-generating process less disk-dependent (and possibly make it k8s-ready), I propose we serve these artifacts directly from swift. However, it'd be nice to bring on some tool so we do not have to write a webapp to generate the directory lists in house. I'm imagining something like Apache's DirectoryListings rendered html list of the objects in swift and proxy users to download the object from object storage directly. @MatthewVernon indicated that hosting a page like this with S3 might be easy. I've no preference on object store flavor, but with slight edge to S3 as it has wider support among ETL processors I've looked at thus far. TL;DR - Does Data Persistence support anything that we could proxy to that could serve as an "indexed directory" listing of artifacts stored in and served from a bucket?

Currently, we host flamegraph SVGZ and compressed PHP Excimer logs as an openweb-accessible directory from a single host per-dc from performance.wm.o. We replicate these artifacts to Swift with a 3-year TTL set on them. The hosts backing this open directory have smaller disks than what we keep, requiring us to manually manage the artifacts which can balloon unexpectedly under periods of extra load. To support the possibility of making this flamegraph-generating process less disk-dependent (and possibly make it k8s-ready), I propose we serve these artifacts directly from swift. However, it'd be nice to bring on some tool so we do not have to write a webapp to generate the directory lists in house. I'm imagining something like Apache's DirectoryListings rendered html list of the objects in swift and proxy users to download the object from object storage directly. @MatthewVernon indicated that hosting a page like this with S3 might be easy. I've no preference on object store flavor, but with slight edge to S3 as it has wider support among ETL processors I've looked at thus far. TL;DR - Does Data Persistence support anything that we could proxy to that could serve as an "open"indexed directory" listing of artifacts stored in and served from a bucket?