Page MenuHomePhabricator

alnitak disk space warning
Closed, ResolvedPublic

Description

Alnitak has an open disk space warning in icinga

DISK WARNING - free space: 
/ 3447 MB (49% inode=80%):
/sys/fs/cgroup 0 MB (100% inode=99%):
/dev 32184 MB (99% inode=99%):
/run 6438 MB (99% inode=99%):
/run/lock 5 MB (100% inode=99%):
/run/shm 32196 MB (100% inode=99%): 
/run/user 100 MB (100% inode=99%):
/boot 182 MB (72% inode=99%):
/srv 1011695 MB (20% inode=99%):

Event Timeline

@herron Thanks for the heads up! Just curious, how did you notice that? Thought Jeff and I were the only ones to get alerted about frack stuff anymore.

I am a little confused as I logged into these boxes and got totally different numbers than I see in Icinga:

 1% /dev
 1% /run
51% /
 0% /sys/fs/cgroup
 0% /run/lock
 0% /run/shm
 0% /run/user
28% /boot
80% /srv

Hey @cwdent, I just noticed the disk alert listed in the icinga "unhandled services" board and created this to acknowledge it.

So the icinga numbers reflect free space while du shows used space. I think the culprit is /srv with 80% used (or 20% free from icinga's perspective)

@herron ah yes of course, pardon my brain fart :P

We are aggregating kafka data on that disk right now so it is extra full, but will be freeing it up again within a day or so. Thanks for acking, I think that is fine for now.

Jgreen claimed this task.
Jgreen subscribed.

We cleaned up and compressed files on the nfs mount, which dropped it back to 67%