Per parent ticket, a recent thanos-swift cluster upgrade caused the RDF Streaming Updater snapshot storage usage to skyrocket (from 200 GB -> 2 TB). Since the Cirrus Streaming Updater uses thanos-swift as well, @RKemper checked its storage usage and found it to be excessively high as well (~13 TB). We had to check this manually, as there are currently no alerts or dashboard panels associated with the Cirrus Streaming Updater's thanos-swift storage usage.
Creating this ticket to:
- Create alerts for the Cirrus Streaming Updater's thanos-swift storage usage.
- Add a panel to the Cirrus Streaming Updater's dashboard that shows thanos-swift storage usage: here
- Set up S3 client config on stat hosts (see this patch and T432167 for an idea of what we might do), so we can interact with our swift accounts without having to dig up credentials and set up a client manually every time.