Notification Type: PROBLEM Service: MariaDB disk space Host: db1047 Address: 10.64.16.36 State: CRITICAL Date/Time: Sun Dec 18 06:25:53 UTC 2016 Additional Info: DISK CRITICAL - free space: / 398 MB (5% inode=54%):
Description
Description
Event Timeline
Comment Actions
We believe this host alerted for "/" and not for the /srv/ partition itself.
There are not massive queries running at that time which could cause big temporary tables.
But the "/ partition on the other hand looks pretty small and it is heavily used:
marostegui@db1047:~$ df -hT Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 32G 4.0K 32G 1% /dev tmpfs tmpfs 6.3G 748K 6.3G 1% /run /dev/sda1 ext3 7.3G 5.9G 990M 86% / none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup none tmpfs 5.0M 0 5.0M 0% /run/lock none tmpfs 32G 0 32G 0% /run/shm none tmpfs 100M 0 100M 0% /run/user /dev/mapper/tank-data xfs 6.6T 5.1T 1.5T 78% /srv
There are some logs there that can probably be delete and give give us some space back
root@db1047:/var/log# du -sh account/ atop/ 1.1G account/ 507M atop/
@fgiunchedi we saw you logged in around the time of the page, where you able to see what exactly grew to reach the limit?
Comment Actions
Correct, the page was for /. There were some old atop.log log files from 2015 I've removed and cleared the apt cache. Other than that the host seems to be creeping up disk space anyways over time. IIRC it was mentioned that it is mostly due to process accounting as @Marostegui showed above
Comment Actions
Thanks for the input @fgiunchedi!
In order to avoid it paging during the holidays period this is what I have done:
root@db1047:/var/log# mkdir /srv/tmp/log_bkup root@db1047:/var/log# cp -r atop/ account/ /srv/tmp/log_bkup/
And truncated all those logs from /var/log
root@db1047:~# df -hT / Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext3 7.3G 4.7G 2.2G 69% /