The ArcLamp pipeline collects stack traces (via Redis) and produces the flame graphs seen at https://performance.wikimedia.org/php-profiling/.
Right now, the processed logfiles and SVGs are stored in /srv/xenon on webperf1002, and processing runs via cron on that host. We would like to run the pipeline in a more distributed manner (T227026), increase retention (T200108), and not require bespoke backup/restore/failover procedures for this data. I believe the path forward is thus to store this data in Swift. The analytics cluster was also considered, however the data needs to be externally available via HTTPS on the performance site.
I've done some initial work towards rewriting the cron job to read/write data from a local Swift instance on my laptop. I would like to start running this on real data, initially in parallel with the current pipeline.
This task is to determine replication, etc. settings and create Swift container(s) for this data. As I do not have admin rights in the Swift cluster, some SRE input & assistance is requested.