thanos-be1001 disk space alerts
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	Jan 20 2024, 5:50 AM

Description

This alert has been going on for days.
I've acked it

06:41:34 <jinxer-wm> (DiskSpace) firing: (6) Disk space thanos-be1001:9100:/srv/swift-storage/sdd1 5.73% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace

Related Objects

Mentioned Here: T351927: Decide and tweak Thanos retention

Event Timeline

Marostegui created this task.Jan 20 2024, 5:50 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 20 2024, 5:50 AM

Happens with other hosts too

@fgiunchedi are you able to look at thanos retention again, please? [I think T351927 is related].

Yes I'll be taking a look at what we can drop in terms of replicated data

I've followed up in T351927 and freed some space, so we're back to non-alerting levels. I'll be discussing with o11y a more comprehensive solution for longer term sustainability

thanos-be1001 disk space alertsClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

thanos-be1001 disk space alerts
Closed, ResolvedPublic
Actions