Hi, sorry this got dropped - do feel free to poke.

Can you give me an idea of number & size of objects, and what sort of bandwidth you expect this to need, please? I'll make a note to discuss at our next data persistence meeting.

Other than upload.wikimedia.org (which is backed by swift), our object storage is generally only internally-visible at present. There are little utilities for putting a web frontend on a S3 bucket which is a bit prettier than the Ceph default e.g. https://github.com/flightlesstux/S3-Directory-Listing but traversing the CDN is still going to need some thought.

In T413127#11698730, @MatthewVernon wrote:

Can you give me an idea of number & size of objects, and what sort of bandwidth you expect this to need, please?

The files I'd like to serve can be found in swift at https://ms-fe.svc.eqiad.wmnet/v1/AUTH_performance/arclamp-(logs|svgs)-(hourly|daily)
Some napkin math and a 3-year retention period (currently configured) yields 2,023,560 files. and ~634GB of data. Files range from a few hundred bytes to a little over a hundred megabytes.

There are little utilities for putting a web frontend on a S3 bucket which is a bit prettier than the Ceph default e.g. https://github.com/flightlesstux/S3-Directory-Listing but traversing the CDN is still going to need some thought.

This little utility is basically what we're looking for. We (Observability) could host something like it, but it'd be neat if this index was a service we could proxy or redirect traffic towards.

2M files can be too much for swift for one container and generally the backends. I have some suggestions:

• Maybe drop all hourly graphs and logs after three months?
• We could also drop all non "all" graphs and logs after a year too?

I think dropping those would save a lot of space without dropping too much useful stuff.

In T413127#11715148, @Ladsgroup wrote:

2M files can be too much for swift for one container and generally the backends. I have some suggestions:

• Maybe drop all hourly graphs and logs after three months?
• We could also drop all non "all" graphs and logs after a year too?

I think dropping those would save a lot of space without dropping too much useful stuff.

I agree, however I'll defer to @Krinkle as one of the main users of this data.

Some history:
T200108: Increase retention of ArcLamp SVGs to 2 years
T244776: Swift container for performance flame graphs (ArcLamp)
https://gerrit.wikimedia.org/r/plugins/gitiles/performance/arc-lamp/+/refs/heads/master/arclamp-compress-logs#20

We replicate these artifacts to Swift with a 3-year TTL

Arc Lamp SVGs are not meant to have any expiry, and afaik don't (and should not) have any expiry or scheduled object removal.

The flame graphs are only a few KB each and we keep them indefinitely since March 2021. The interface at https://performance.wikimedia.org/php-profiling/ reflects this, and files like https://performance.wikimedia.org/arclamp/svgs/daily/2021-03-29.excimer.index.svgz continue to work today.

The 3 year expiry in arclamp-compress-logs is specifically for .log.gz files, not .svg.

Maybe drop all hourly graphs and logs after three months?

Sounds good to me.

We could also drop all non "all" graphs and logs after a year too?

I don't think we should vary retenetion between individual entrypoints and the "all" entrypoint. This makes the tool harder to reason about, and would complicate the UI. We use both both equally depending on the task or use case, and afaik those use cases don't vary much between investigations for recent and longer-running regressions. We need the "all" entrypoint either way.

I suggest we tune retention only across type (log vs svg) and period (daily vs hourly).

Specifically, I suggest:

• Daily SVGs: indefinite (unchanged).
• Daily logs: 3 years -> 3 months
• Hourly SVGs: indefinite -> 3 months.
• Hourly logs: 3 years -> 3 months.

This should cut overall storage and file count by ~90% (36 months vs 3 months), and reduce daily growth in file count by 98% (from 50N/day to 1N/day); where 50N = 25 periods per day (24h + 1daily), times 2 file types (svg+log), times N entrypoints; and where 1N = 1 period (daily) in 1 format (svg).

I assume that absolute storage size is fairly stable with negligible absolute growth there, given that the SVGs are tiny and the log files already have an expiry.

Sounds good to me. I try to implement it ASAP.

Change #1271095 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[performance/arc-lamp@master] Decrease time to live for the arclamp logs to three months

https://gerrit.wikimedia.org/r/1271095

Change #1271095 merged by jenkins-bot:

[performance/arc-lamp@master] Decrease time to live for the arclamp logs to three months

https://gerrit.wikimedia.org/r/1271095

Mentioned in SAL (#wikimedia-operations) [2026-04-20T14:45:53Z] <cwhite@deploy1003> Started deploy [performance/arc-lamp@bd7b2ab]: T413127

Mentioned in SAL (#wikimedia-operations) [2026-04-20T14:46:02Z] <cwhite@deploy1003> Finished deploy [performance/arc-lamp@bd7b2ab]: T413127 (duration: 00m 08s)