Page MenuHomePhabricator

tin disk space at 5%
Closed, ResolvedPublic

Description

Tin started sending an alert of low disk space on Icinga (there is still 14G available). I am assigning this to the release engineering team (with ops assistance) to see if some large change on the files there was done recently? In many cases this is logs or any temporary structure and can be easily deleted.

If you do not know, I can log in myself and find what is the main usage, but preferred to notify you first, in case it was something known by you.

If the extra space used was actually needed because of requirement growth, we would need to see how to provide it.

Event Timeline

jcrespo raised the priority of this task from to Needs Triage.
jcrespo updated the task description. (Show Details)
jcrespo added a subscriber: jcrespo.
jcrespo set Security to None.
jcrespo updated the task description. (Show Details)

The thing that takes all the space here mostly is /var/lib/l10nupdate/caches is 53G.

This seems familiar, like we had the same issue before, and the question is if we need to keep all these versions around:

cache-1.25wmf1   cache-1.25wmf15  cache-1.25wmf20  cache-1.25wmf4  cache-1.26wmf1   cache-1.26wmf15  cache-1.26wmf20  cache-1.26wmf6
cache-1.25wmf10  cache-1.25wmf16  cache-1.25wmf21  cache-1.25wmf5  cache-1.26wmf10  cache-1.26wmf16  cache-1.26wmf21  cache-1.26wmf7
cache-1.25wmf11  cache-1.25wmf17  cache-1.25wmf22  cache-1.25wmf6  cache-1.26wmf11  cache-1.26wmf17  cache-1.26wmf22  cache-1.26wmf8
cache-1.25wmf12  cache-1.25wmf18  cache-1.25wmf23  cache-1.25wmf7  cache-1.26wmf12  cache-1.26wmf18  cache-1.26wmf3   cache-1.26wmf9
cache-1.25wmf13  cache-1.25wmf19  cache-1.25wmf24  cache-1.25wmf8  cache-1.26wmf13  cache-1.26wmf19  cache-1.26wmf4
cache-1.25wmf14  cache-1.25wmf2   cache-1.25wmf3   cache-1.25wmf9  cache-1.26wmf14  cache-1.26wmf2   cache-1.26wmf5

@Krenair is also using tin to upload files (~92G) - T111941: Please upload large files to Wikimedia Commons because terbium doesn't have enough space.

I deleted the 46GB original archive, other files will go soon.

Original report of <6% disk space alert is gone. I am retiring operations from the projects (sadly, we do not have a "Done" column), and let tin's users full control of it (close the ticket, delete more files, request more resources, etc.).

Filled in a few other tasks:

T112509: Trebuchet should repack / pack-refs git repos under /srv/deployment
T112508: scap / deployment of branches should get rid of old caches on tin /var/lib/l10nupdate/caches

What about those /tmp/mw-cache-* files?

Deleted the remaining upload files (since that's resolved now) - 106G (42%) free

hashar claimed this task.

If one has any garbage collection ideas, feel free to create subtasks as appropriate.

Since @Krenair cleaned up all disk and we are now at 42% free, I am closing this task.