A great many media files have thumbnails generated on upload which are not accessed again (see graphs below)
So ...
When we're generating a thumbnail
If we set a TTL on the generated file in Swift, if and only if it has been uploaded recently
Then we might hope to see large reduction in thumbnail growth in return for a small increase in load on the thumbnail generation engine
The code
There's already some code around expiring objects in Swift in wikimedia_thumbor/result_storage/swift/swift.py, see SWIFT_THUMBNAIL_EXPIRY_SECONDS
Here's what Claude suggests
1. Capture the original media file's age
wikimedia_thumbor/loader/swift/__init__.py, in load() right after the successful get_object
# Swift returns the object's creation time as x-timestamp (epoch float,
# lowercased by swiftclient). Stash it for the result storage's TTL decision.
try:
context.wikimedia_original_timestamp = float(headers.get('x-timestamp'))
except (TypeError, ValueError):
context.wikimedia_original_timestamp = None2. Store the result with a conditional expiry
wikimedia_thumbor/result_storage/swift/swift.py, in put() where the existing SWIFT_THUMBNAIL_EXPIRY_SECONDS block sits
expiry = self.context.config.get('SWIFT_THUMBNAIL_EXPIRY_SECONDS', 0)
jitter = self.context.config.get('SWIFT_THUMBNAIL_EXPIRY_JITTER_SECONDS', 0)
age_threshold = self.context.config.get(
'SWIFT_THUMBNAIL_SPECULATIVE_AGE_SECONDS', 0)
containers = self.context.config.get(
'SWIFT_THUMBNAIL_EXPIRY_CONTAINERS', [])
original_ts = getattr(self.context, 'wikimedia_original_timestamp', None)
if (
expiry > 0
and age_threshold > 0
and original_ts is not None
and self.context.wikimedia_thumbnail_container in containers
and time.time() - original_ts < age_threshold
):
headers['X-Delete-After'] = str(int(expiry + random.uniform(0, max(jitter, 0))))Claude also says "Config lands wherever the other SWIFT_* values live in the thumbor deployment chart" ... I guess that's in puppet/modules/netbox/templates/configuration.py.erb but I don't know
SWIFT_THUMBNAIL_EXPIRY_CONTAINERS is a list of containers that are gonna be the experiment containers. Probably we should pick just one, and it needs to be one that hasn't had the delete script run recently (within the last month or pref two or more) so that it's not getting a big growth spike from that.
SWIFT_THUMBNAIL_SPECULATIVE_AGE_SECONDS is the threshold for the age of original - if an original is younger than this number of seconds then we set a TTL. Looking at the graphs below I'd say we try this at 86400 seconds (i.e. 1 day) first
SWIFT_THUMBNAIL_EXPIRY_SECONDS is the short TTL we'll set for new uploads. Looking at the graphs I think it's worth trying this at 172800 seconds (i.e. 48 hours) first
SWIFT_THUMBNAIL_EXPIRY_JITTER_SECONDS is so that if a bunch of thumbnails get created at the same time they don't all expire at the same time and cause a thumbor load spike. Honestly I'm not sure if we need this, might be simpler to leave it out for now (or leave it in, or set it to zero, I don't mind!)
We can turn the experiment off by setting SWIFT_THUMBNAIL_EXPIRY_CONTAINERS to empty or SWIFT_THUMBNAIL_SPECULATIVE_AGE_SECONDS to zero, but note that any TTLs set will still be in force, and the experiment won't fully end until anything with a TTL has expired.
Some unit tests would be really good here btw
Testing
Once the above is done and merged it's probably a good idea just to run a manual test with a new upload and check its X-Delete-After
Monitoring
- thumbnail growth in the experimental container versus control container(s) (again pick one that hasn't had the delete script run on it recently) - expecting it to be significantly reduced (75%)
- proportion of generated / stored-in-object-store thumbnails in the experimental container versus control container(s) - expecting generated to grow, but not by a lot (<10%, should not increase over time)
- p50/p75/p95/p99 response times in the experimental container versus control container(s) - expecting p95 and p99 to grow, but only by a little
- in grafana keep an eye on rate(thumbor_swift_thumbnail_exception_count{operation="write"}[5m]) to make sure the new code isn't causing issues with swift
Note that we won't see any changes in thumbnail growth or thumbor load or latency until AFTER the first newly-created thumbnails TTLs start to expire.
Some graphs for uploads on May 1 2026
